Scheduled regression test

Problem: Nobody adds a performance regression on purpose. Regressions arrive as side effects of feature changes. Without a recurring test that compares today’s result to yesterday’s, they stay hidden until a customer reports them. A scheduled regression test shows drift automatically, every morning before standup.

Test type: Baseline regression test, run on a cron schedule.

Prerequisites

• A MaxoPerf test that has produced at least one clean baseline run (with p95 latency and error rate within your SLO).
• A cron expression for the schedule. See the Cron quick reference in the how-to.

Step by step in MaxoPerf

1. Designate a baseline run

1. Open the test and switch to the Runs tab.
2. Find the most recent run that met your SLO (low error rate, p95 below threshold).
3. Note the run ID. You need it later when you set up the comparison.

2. Set the failure criteria

A scheduled test works as a regression gate only if it marks a run as failed on its own when performance degrades. Configure failure criteria now. See Failure criteria pass/fail gates for the full recipe.

Quick version:

1. Open the test, switch to Configuration.
2. Scroll to Failure criteria and click Add criterion.
3. Add: p95 latency > 500ms (replace with your SLO threshold).
4. Add: error rate > 1 %.
5. Save.

3. Create the nightly schedule

1. Switch to the Schedules tab.
2. Click New schedule.
3. Enter a cron expression, for example 0 3 * * * (every night at 3 AM).
4. Pick your timezone (schedule times are timezone-aware).
5. Name it nightly-regression and enable it.
6. Save.

4. Check the morning results

1. Open the test’s Runs tab. The scheduled run appears at the top.
2. A green Finished badge means the run passed the failure criteria.
3. A red Failed badge means at least one failure criterion was violated. Click the run to see which criterion failed and by how much.

5. Compare to the baseline

1. Open the latest scheduled run.
2. Click Compare in the run header.
3. Select the designated baseline run from step 1.
4. The comparison view shows delta chips for each key metric: green for improvement, red for regression.

Rotate the baseline periodically

After two clean weeks of scheduled runs, promote the most recent clean run as the new baseline. Your team then stops chasing regressions against an old (and possibly slow) baseline.

Verify

• The run appears in the Runs list at the expected time (within a few minutes of the cron fire time).
• Runs that exceed the failure criteria show Failed status, not Error. An error status means the run itself did not start or crashed. It does not mean a performance threshold was violated.
• The comparison view loads when you select the baseline run.

Variations

• Weekly soak baseline: combine a weekly schedule with a longer hold-for. See Overnight soak test.
• CI + scheduled dual coverage: use CI-triggered runs for every deploy and a nightly scheduled run for the baseline comparison. See CI-gated performance test.
• Slack/webhook notification: configure a webhook on run completion so the team gets a Slack message when a scheduled run fails.

Where to go next

• Failure criteria pass/fail gates: the full failure-criteria recipe.
• Comparing runs and baselines: a closer look at the comparison view.
• Schedule recurring tests: the how-to reference for scheduling.
• Baseline regression test: what this test type measures.