Skip to content

Updated June 15, 2026

Regression and baseline testing

Establish performance baselines before changes ship and catch regressions automatically by comparing every run against your stored reference baseline.

What this workflow should produce

  • A named baseline run that stores the "known-good" performance state for each service or API.
  • Automatic release-over-release comparison that flags regressions above a configurable tolerance band.
  • An attributable regression signal: the specific label, metric, and delta that caused the failure.
  • A historical record of performance evolution that supports capacity planning and incident retrospectives.

Use case

Threshold-based gates catch dramatic performance regressions — a 300 ms endpoint suddenly taking 1.2 s. They are less effective at catching gradual drift: an endpoint that takes 280 ms today, 290 ms next week, 310 ms the week after. Each individual change is within tolerance, but over three releases the service has regressed 10% in a way that no single gate caught.

Baseline testing addresses the gradual drift problem. The approach is simple: establish a named “known-good” run under stable conditions, compare every subsequent run against that baseline, and flag any run where the delta exceeds a configurable tolerance. The comparison is relative (percentage change), not absolute, so it catches regressions before they cross an SLO threshold.

The discipline also creates a record of intentional performance improvements. When a query optimisation drops p95 from 280 ms to 180 ms, re-baselining at the new performance level means the next run will be compared against the improved standard, not the pre-optimisation baseline. Performance improvements are locked in, not eroded by subsequent changes.

How MaxoPerf supports it

Every MaxoPerf run is stored with its configuration, results, and labels. Runs can be tagged and named (e.g. baseline-v3.0.0-prod-equivalent). The compare view lets you select any two runs and see the per-label delta — absolute values and percentage change — for every metric.

For automated regression detection, the MaxoPerf API lets CI workflows compare the current run against a designated baseline run ID and fail if any label’s metric delta exceeds a configured percentage. This integrates into the same release pipeline as threshold gating, adding a relative check alongside the absolute check.

Run history is cumulative. Over months, the stored run history shows the performance trajectory of each labeled endpoint — not just whether a run passed, but whether the service is consistently trending towards or away from its performance targets.

A concrete baseline discipline

A backend team at a payments company maintains a baseline for their transaction-processing API:

  • Baseline established: after every major performance improvement, a manual run is executed against a production-equivalent staging environment, tagged baseline-<version>-<date>, and recorded as the reference for the next release cycle
  • Current baseline: baseline-v4.1.0-2026-05-10, with p95 of 140 ms for POST /v2/transactions, error rate 0.03%
  • CI gate: every RC run compares against the current baseline; a delta > 15% on any label’s p95, or > 50% on error rate, fails the pipeline with the specific label and delta reported
  • Re-baseline trigger: when a PR explicitly improves performance (new index, query optimisation, caching), the engineer runs the baseline script after merge to production and updates the reference
  • Gradual regression caught: over a 2-month period the team noticed the compare view showing POST /v2/transactions p95 trending from 140 ms toward 170 ms across 8 releases, each within the 15% tolerance individually. Investigating the cumulative drift identified a query plan change introduced by an ORM version bump; rolling back the ORM version restored p95 to 138 ms.

Practical rollout

  1. Establish the first baseline explicitly — do not assume the first run is a valid baseline. Run the test 5+ times, verify results are consistent, and tag the baseline with a meaningful name and date.
  2. Re-baseline after intentional improvements — if you merge a performance optimisation, update the baseline. A baseline that does not reflect the current best-known performance is a regression magnet.
  3. Use percentage deltas, not absolute values, for the comparison gate — a 15% delta gate adapts to changes in baseline performance level; an absolute-value gate does not.
  4. Combine with threshold gating — baseline comparison catches drift; threshold gating catches SLO violations. Run both checks in the same CI gate step.
  5. Review the run-history trend monthly — look at the last 4–8 runs for each labeled endpoint. A service where every run is within tolerance but the average is slowly increasing is a service that will breach its threshold in 2–3 more releases. Intervene early.

FAQ

How do I establish a valid baseline when my service performance fluctuates?

Run the test at least 5 times under stable conditions (no application changes, no infrastructure maintenance) and use the median result as the baseline. Avoid using a single run, which may be an outlier. Tag the baseline with a version label and the date it was established. Re-baseline after any intentional performance improvement so the baseline reflects the current known-good state, not a historical state that has since been improved.

What tolerance band should I use for regression detection?

A common starting point is a 10–15% tolerance on p95 latency and a 20% tolerance on error rate. Tighter tolerances produce more alerts; looser tolerances miss subtle regressions. Tune the band based on the natural variance you observe across baseline runs — if p95 fluctuates ±8% run-to-run on a healthy service, a 10% regression threshold will produce frequent false positives.

How is baseline testing different from threshold-based gating?

A threshold gate uses absolute values (p95 must be below 300 ms). A baseline comparison uses relative values (p95 must not have increased more than 15% compared to the last good baseline). Both are valuable: thresholds catch violations of your SLO commitments; baseline comparisons catch gradual regressions that are still within absolute thresholds but trending in the wrong direction. Use both together for robust regression detection.

Related documentation

Related use cases

  • AI inference load testing

    Validate LLM inference APIs under concurrent request load — measure time-to-first-token, token throughput, and cost per request to right-size GPU capacity.

  • Frontend browser performance testing

    Load-test real browser journeys — login flows, SPAs, and media-heavy pages — measuring user-perceived latency and render performance under concurrency.

  • Microservices SLO validation

    Confirm each service meets its latency and error-budget SLOs under realistic request concurrency before changes are promoted to production environments.

Run your first test Compare plans