Baseline regression test

A baseline regression test is not a test type of its own. It is a practice: you run the same load profile on every build and compare the result automatically against a known-good baseline. If performance degrades, the build fails. That is how CI/CD catches performance regressions before they reach production.

Before you start

• You have a load test with a stable, passing result that you designate as the baseline.
• Your CI pipeline can trigger MaxoPerf runs via the API or CLI. See CI-gated performance test.
• You have defined quantified thresholds: for example, “p95 latency must not increase by more than 20 % from baseline” or “throughput must not drop by more than 10 %”.

What is a baseline regression test?

It has two parts:

1. Baseline. A reference run (or set of runs) with acceptable performance, taken from a known-good state of the codebase.
2. Regression gate. Each new run is compared to the baseline. If any metric goes past the allowed delta, the run is marked failed and the CI job fails.

In MaxoPerf you build this with failure criteria on the run, plus the run comparison view for manual review.

How to set up a baseline regression test in MaxoPerf

Step 1: Define and freeze the baseline run

1. Run your standard load test on the main branch in a known-good state.
2. Open the run result and click Mark as baseline (or add a baseline tag + a note that says what this run represents).
3. Record the run ID and key metric values (p95 latency, throughput, error rate). Store these in your CI configuration or team handbook.

Step 2: Configure failure criteria as the regression gate

1. Open the test definition and navigate to the Failure criteria section.

2. Add thresholds that express your regression budget. Example:

   Metric	Condition	Threshold
   p95 latency	less than	500 ms
   error rate	less than	1 %
   throughput	greater than	80 RPS

3. Save the test. Any later run that breaks these thresholds is marked failed, and that status can gate a CI pipeline step.

Step 3: Trigger the test from CI

In your CI pipeline (GitHub Actions, GitLab CI, Jenkins, etc.), add a step that:

1. Triggers a MaxoPerf run via the public API (see Run your first test via API).
2. Polls for the run to reach finished or failed status.
3. Exits with a non-zero code if the run status is failed.

A minimal GitHub Actions step:

- name: Run performance regression gate
  run: |
    RUN_ID=$(curl -s -X POST "$MAXOPERF_API/v1/runs" \
      -H "Authorization: Bearer $MAXOPERF_TOKEN" \
      -d '{"testId":"${{ env.PERF_TEST_ID }}"}' | jq -r '.id')
    echo "Run started: $RUN_ID"

    for i in $(seq 1 60); do
      STATUS=$(curl -s "$MAXOPERF_API/v1/runs/$RUN_ID" \
        -H "Authorization: Bearer $MAXOPERF_TOKEN" | jq -r '.status')
      [ "$STATUS" = "finished" ] && exit 0
      [ "$STATUS" = "failed" ]   && exit 1
      sleep 30
    done
    echo "Timed out waiting for run" && exit 1
  env:
    MAXOPERF_TOKEN: ${{ secrets.MAXOPERF_TOKEN }}

How to read the result

For each build:

• Status badge. finished means all failure criteria passed. failed means a threshold was breached.
• Comparison view. Open the run and compare it against the pinned baseline. Check which metric crossed the threshold and by how much.
• Trend. Across several builds, look for gradual drift: each build passes on its own, but together they add up to a regression.

Tip

Update the baseline after an intentional performance improvement. If a refactor drops p95 from 350 ms to 200 ms, re-baseline. The threshold gets tighter, and you catch future regressions earlier.

Do / don’t

Do	Don’t
Pin a specific run ID as the baseline	Compare against “the last run”, which could already be a regression
Set thresholds based on your SLO budget, not aspirational values	Set thresholds so tight that every run fails and the team ignores them
Update the baseline after intentional improvements	Leave a stale baseline that makes every run look great
Tag runs with the commit SHA for traceability	Run the regression test outside of CI without commit context

Where to go next

• Configuration test: compare configuration variants as well as builds.
• Foundations: Baselines, SLOs, and error budgets: how to set meaningful thresholds.
• Cookbook: CI-gated performance test: full end-to-end CI integration with a pass/fail gate.
• Cookbook: Comparing runs and baselines: how to use the MaxoPerf run comparison view.