Comparing runs and baselines

Problem: A run result in isolation does not tell you whether performance has improved or degraded. Comparing two runs — a current run against a designated baseline — produces the delta metrics that answer the only question that matters: is it better or worse than last time?

Test type: Baseline regression test. Also useful after any test type change.

Prerequisites

• At least two completed runs of the same test (or two runs of tests with the same request labels).
• A designated baseline run — typically the last run that met your SLO before the current change.

What the comparison view shows

The MaxoPerf comparison view places two runs side by side and computes delta chips for:

• p50, p90, p95, p99 latency per label and aggregate.
• Throughput (RPS) — aggregate and per label.
• Error rate — aggregate and per label.
• VU count (to verify the test conditions were the same).

Delta chips are green for improvement and red for regression, relative to the baseline run.

Step by step in MaxoPerf

1. Open the current run

1. Go to Runs in the console and click the run you want to evaluate.
2. The run detail page opens on the Overview tab.

2. Open the comparison view

1. In the run header, click the Compare button (or Compare to baseline if a default baseline is set).
2. A selector appears — choose the baseline run from the list. Recent runs are shown first; use the search box to find by run ID or date.
3. The page reloads in split view with the current run on the left and the baseline on the right.

3. Read the delta chips

Each metric card shows three values:

• Current run value — for example 245 ms p95.
• Baseline value — for example 257 ms p95.
• Delta — −12 ms (−5 %) in green, or +42 ms (+16 %) in red.

Work through the labels:

• Green across all labels — the change is an improvement or neutral. Safe to ship (subject to other checks).
• Red on one label — investigate that endpoint. Was the VU count the same? Was the data different?
• Red on aggregate but mixed per-label — a slow endpoint is dragging the average. Drill in.

Compare apples to apples

The comparison is only meaningful if both runs used the same VU count, the same locations, and the same data. Check the run summary headers to confirm before drawing conclusions.

4. Share the comparison

The comparison view URL is stable — it encodes both run IDs. Copy it from the browser address bar and paste it into a pull request review, Slack message, or incident ticket. Teammates click the link and see the same view.

Verify

• The comparison page shows both run IDs in the header.
• Delta chips appear on all key metrics — if a label is missing from one run (because a scenario changed), the comparison shows N/A for that label.
• The VU count and duration for both runs are visible in the summary section — confirm they match before interpreting delta chips.

Variations

• Set a default baseline: in the test Configuration tab, pin a run as the “official” baseline. Every new run will show a delta against it automatically on the overview tab without needing to open the compare view.
• Cross-test comparison: compare a run from a new Taurus scenario against the equivalent k6 run to validate that the migrated scenario produces the same throughput.
• Post-incident comparison: after a production incident, compare the last pre-incident run to the first post-incident run to understand whether performance changed alongside the event.
• VarioTest child run comparison: open a VarioTest child run and compare it to the same scenario from a prior VarioTest to isolate per-scenario regressions.

Where to go next

• Failure criteria pass/fail gates — encode the comparison result into an automatic pass/fail verdict.
• Build a reporting dashboard — persist the key metrics comparison as a saved chart layout.
• Baseline regression test — the test type this recipe exercises.
• Baselines, SLOs, and error budgets — how to choose and maintain the right baseline.