Baseline regression test

A baseline regression test is not a new test type in itself — it is a practice: you run the same load profile on every build and automatically compare the result against a known good baseline. If performance degrades, the build fails. This is how performance regressions are caught in CI/CD before reaching production.

Before you start

• You have a load test with a stable, passing result that you designate as the baseline.
• Your CI pipeline can trigger MaxoPerf runs via the API or CLI. See CI-gated performance test.
• You have defined quantified thresholds: for example, “p95 latency must not increase by more than 20 % from baseline” or “throughput must not drop by more than 10 %”.

What is a baseline regression test?

The concept has two parts:

1. Baseline — a reference run (or set of runs) that represents acceptable performance. The baseline is taken from a known-good state of the codebase.
2. Regression gate — each new run is compared to the baseline. If any metric exceeds the allowed delta, the run is marked as failed and the CI job fails.

In MaxoPerf, this is implemented with failure criteria on the run and the run comparison view for manual review.

How to set up a baseline regression test in MaxoPerf

Step 1 — Define and freeze the baseline run

1. Run your standard load test on the main branch in a known-good state.
2. Open the run result and click Mark as baseline (or add a baseline tag + a note documenting what this run represents).
3. Record the run ID and key metric values (p95 latency, throughput, error rate). Store these in your CI configuration or team handbook.

Step 2 — Configure failure criteria as the regression gate

1. Open the test definition and navigate to the Failure criteria section.

2. Add thresholds that encode your regression budget. Example:

   Metric	Condition	Threshold
   p95 latency	less than	500 ms
   error rate	less than	1 %
   throughput	greater than	80 RPS

3. Save the test. Any future run that violates these thresholds will be marked failed, which can gate a CI pipeline step.

Step 3 — Trigger the test from CI

In your CI pipeline (GitHub Actions, GitLab CI, Jenkins, etc.), add a step that:

1. Triggers a MaxoPerf run via the public API (see Run your first test via API).
2. Polls for the run to reach finished or failed status.
3. Exits with a non-zero code if the run status is failed.

A minimal GitHub Actions step:

- name: Run performance regression gate
  run: |
    RUN_ID=$(curl -s -X POST "$MAXOPERF_API/v1/runs" \
      -H "Authorization: Bearer $MAXOPERF_TOKEN" \
      -d '{"testId":"${{ env.PERF_TEST_ID }}"}' | jq -r '.id')
    echo "Run started: $RUN_ID"

    for i in $(seq 1 60); do
      STATUS=$(curl -s "$MAXOPERF_API/v1/runs/$RUN_ID" \
        -H "Authorization: Bearer $MAXOPERF_TOKEN" | jq -r '.status')
      [ "$STATUS" = "finished" ] && exit 0
      [ "$STATUS" = "failed" ]   && exit 1
      sleep 30
    done
    echo "Timed out waiting for run" && exit 1
  env:
    MAXOPERF_TOKEN: ${{ secrets.MAXOPERF_TOKEN }}

How to read the result

For each build:

• Status badge — finished means all failure criteria passed; failed means a threshold was breached.
• Comparison view — open the run and compare it against the pinned baseline. Check which metric crossed the threshold and by how much.
• Trend — over multiple builds, look for gradual drift that individually passes each threshold but represents a creeping regression.

Tip

Update the baseline after intentional performance improvements. If a refactor drops p95 from 350 ms to 200 ms, re-baseline so the new threshold is tighter and future regressions are caught earlier.

Do / don’t

Do	Don’t
Pin a specific run ID as the baseline	Compare against “the last run” — it could already be a regression
Set thresholds based on your SLO budget, not aspirational values	Set thresholds so tight that every run fails and the team ignores them
Update the baseline after intentional improvements	Leave a stale baseline that makes every run look great
Tag runs with the commit SHA for traceability	Run the regression test outside of CI without commit context

Where to go next

• Configuration test — compare across configuration variants, not just build-to-build.
• Foundations: Baselines, SLOs, and error budgets — how to set meaningful thresholds.
• Cookbook: CI-gated performance test — full end-to-end CI integration with a pass/fail gate.
• Cookbook: Comparing runs and baselines — how to use the MaxoPerf run comparison view.