CI/CD performance gates

Use case

A performance gate turns a repeatable load test into an automated release decision. Without it, performance review is a manual step that gets skipped under deadline pressure, and regressions reach production as user-reported incidents.

The point of a gate is not to create friction. It is to automate the question every release manager already asks informally: “did this change make anything slower or less reliable?” A well-designed gate answers that question in minutes with a binary pass/fail that references specific numbers, specific labels, and a specific run record.

A gate that fails for unexplained reasons, or that triggers on infrastructure noise rather than application behaviour, is worse than no gate — teams learn to click “retry” or “override” reflexively. The design goal is a gate that fails rarely, fails for clear reasons, and that teams trust enough to act on.

How MaxoPerf supports it

MaxoPerf stores the test script, load profile, threshold configuration, run results, and reviewer notes in one place and exposes all of it through a public API. That makes it straightforward to wire release workflows around the same surface your team uses for manual review.

The API lets CI pipelines create runs, poll for completion, and retrieve a structured pass/fail signal with threshold-breach details per label. A breach response looks like: {"label": "POST /v2/cart/checkout", "metric": "p95_ms", "threshold": 300, "actual": 412, "status": "failed"} — enough for a pipeline to print a useful error message and block the promotion without human intervention.

Run history is persistent. Because every gate run is stored with the same metadata (branch, commit SHA, environment, load profile, thresholds), a team investigating a gradual latency regression can compare runs across releases without building their own time-series infrastructure.

A concrete gate configuration

A backend platform team at a SaaS company gates every release candidate to their staging environment:

• Test: a 10-minute scenario covering their five highest-traffic API paths, labelled by endpoint
• Load profile: 60 virtual users, 2-minute ramp, 6-minute hold, 2-minute ramp-down
• Thresholds: p95 < 250 ms per label; error rate < 1%; assertion pass rate > 99%; runner CPU < 80%
• CI integration: a GitHub Actions job calls POST /v1/runs on release-candidate push, polls every 30 seconds, and fails the workflow on any threshold breach
• Override process: a P0 incident or time-critical hotfix can bypass the gate with a run note recording the approver and justification

In the six months since the team introduced this gate, two regressions were caught before production: one a database query that became non-selective after a schema change, and one a third-party dependency that introduced a 200 ms baseline latency on cold-start. Both were caught on the first release-candidate build after the change.

Practical rollout

1. Start with one scenario — pick the API journey your team would most regret discovering was slow in production. Run it manually until it is stable (consistent results run-over-run before any application changes).
2. Set thresholds conservatively — start at 1.5× the current measured p95 and tighten over time. A gate that fails the first three runs due to normal variance will be abandoned.
3. Automate the trigger — use the MaxoPerf API to trigger from CI on release candidates. Start with a non-blocking “observe” mode before enforcing hard failures.
4. Add runner-health guards — configure runner-health thresholds from the start so infrastructure noise does not produce false failures.
5. Expand over time — add scenarios for new API surfaces as they ship, lower thresholds as the service matures, and promote longer soak runs for services where steady-state drift matters.