AI load test failure criteria and run comparison

A MaxoPerf run that ends with status Finished has completed. That does not mean it passed. Without criteria, you have to open every chart by hand to decide whether an LLM or inference endpoint met its SLOs. Failure criteria write your AI SLOs into the test itself. The run then gets an automatic Passed / Failed verdict that CI pipelines, scheduled nightly runs and model-upgrade comparisons can read without a person checking.

Before you start

• Read Cookbook: Failure criteria pass/fail gates. This page adds AI thresholds to that general recipe.
• You have at least one completed baseline run for the LLM endpoint (from LLM performance and load testing).
• You know your SLO targets: p95 TTFT budget, acceptable error rate, minimum throughput floor.

AI-specific failure criteria

Standard load tests gate on p95 latency and error rate. AI endpoints need a few more thresholds:

Criterion	Metric	Typical threshold	Why
TTFT budget	http_req_duration p95 (or llm_ttft_ms custom)	< 800 ms–3 000 ms	Governs UX for chat/copilot features
End-to-end latency	http_req_duration p99	< 10 000 ms	Tail latency, which the slowest users feel
Error rate	Overall error rate	< 2–5 %	Includes 429 rate-limit and 503/504 overload
429 rate-limit rate	Custom rate_limit_rate	< 1 %	Flags quota exhaustion independently of server errors
Throughput floor	Throughput (RPS)	> N RPS	Catches runs where the test itself under-loaded (startup errors)
Custom token metric	llm_ttft_ms p95 (from streaming k6 script)	< 800 ms	TTFT when measured precisely via SSE parsing

Step by step: configuring AI failure criteria

1. Open the test in the MaxoPerf console and go to the Configuration tab.

2. Scroll to Failure criteria and click Add criterion.

3. Add the TTFT / latency criterion:

   • Metric: p95 latency
   • Label: All labels (or chat-completions if you labelled your request)
   • Operator: greater than
   • Value: 3000 (milliseconds; adjust to your model’s expected p95)

4. Add the error-rate criterion:

   • Metric: Error rate
   • Operator: greater than
   • Value: 2 (percent)

5. Add the throughput floor:

   • Metric: Throughput (RPS)
   • Operator: less than
   • Value: 3 (the minimum RPS you expect at your VU count; this stops a misconfigured test from passing)

6. Click Save.

Adding custom metric thresholds

If your k6 script emits a custom metric (e.g. llm_ttft_ms from the Streaming and token latency testing script), MaxoPerf lists it in the custom metrics section:

1. In the Failure criteria section, click Add criterion.
2. Select metric: Custom metric.
3. Enter the metric name: llm_ttft_ms.
4. Aggregation: p95.
5. Operator: greater than.
6. Value: 800.
7. Click Add.

This criterion fails the run if TTFT p95 goes above 800 ms. That is a direct latency SLO for streaming UIs.

Tip

Start with loose thresholds (e.g. p95 < 5 000 ms, error rate < 5 %) for your first few runs to learn the normal range. Tighten them step by step as the baseline settles. AI endpoints are noisier than ordinary REST APIs because generation length varies and GPU scheduling jitters.

Run comparison: model and endpoint versions

When you upgrade a model, change a prompt template or switch inference providers, compare the new endpoint with the baseline run. The comparison puts numbers on every regression and improvement.

How to trigger a comparable run

For a valid comparison, both runs must use:

• The same VU count, ramp-up, and hold duration.
• The same prompt body and max_tokens value.
• The same MaxoPerf runner location.

Create a new test named llm-chat-v2 (or duplicate the baseline test and update the endpoint URL / model name). Run it. Then compare.

Using run comparison in MaxoPerf

1. Open the new run (the candidate).
2. Click Compare in the run detail header. A run-selector dialog appears.
3. Select the baseline run from the same test or a different test with the same scenario.
4. The Overview tab switches to comparison mode. Latency and throughput charts show one line per run, and Δ chips show the difference between runs.

Key delta signals for AI endpoint comparisons:

Δ metric	Interpretation
p95 latency Δ positive	Candidate is slower. Expected if the new model is larger
p95 latency Δ negative	Candidate is faster. Check that output quality has not dropped
Throughput Δ positive	Candidate completes more requests/sec, so inference is more efficient
Error rate Δ positive	Candidate has more errors, from a quota issue or instability
p99 / p95 ratio widens	Candidate has more tail variance: more scheduling jitter or longer outlier generations

Reporting dashboard for AI runs

A MaxoPerf custom dashboard puts the AI endpoint metrics in one view: TTFT percentiles, error rate breakdown and throughput. You stop switching between tabs.

To build an AI-focused dashboard:

1. Open a completed run and click Dashboard → New dashboard.
2. Add a latency percentile panel with p50, p95, p99 lines. Rename it “TTFT / end-to-end latency”.
3. Add a throughput panel and rename it “Inference RPS”.
4. Add an error rate panel segmented by status code (429 vs 5xx).
5. If your script emits llm_output_tokens, add a custom metric panel and set it to llm_output_tokens rate. This is your tokens/sec proxy.
6. Save the dashboard as “AI endpoint standard”. MaxoPerf reuses it on every later run of this test.

CI/CD integration

Failure criteria let you run AI load tests in CI. When a run ends with status Failed, the MaxoPerf API returns the failed status and the criterion that was violated:

# Poll for run completion and surface the verdict
curl -s -H "Authorization: Bearer $MAXOPERF_API_KEY" \
  "https://app.maxoperf.com/api/v1/runs/$RUN_ID" \
  | jq '{status: .status, violations: .failure_criteria_violations}'

A CI job that triggers a run and polls this endpoint can exit non-zero when status == "failed", which blocks the model upgrade deployment without manual steps. See Run a test from CI/CD for the full pipeline guide.

Do / don’t

Do	Don’t
Start criteria loose; tighten over several runs as you learn the normal range	Set criteria tighter than your measured baseline; every run will fail
Use a throughput floor to catch misconfigured tests	Rely on latency criteria alone. A test that sent 1 RPS can pass on latency and prove nothing
Compare runs with identical load parameters for a fair model A/B	Compare runs with different VU counts; latency does not scale linearly with concurrency
Document the SLO thresholds in the test description so reviewers understand the pass bar	Leave criteria values undocumented. Later reviewers won’t know if 3000 ms was chosen on purpose
Use run comparison after every model upgrade, prompt change, or infrastructure change	Review comparisons only after a failure. Comparing every time catches slow regressions

Where to go next

• Cookbook: Failure criteria pass/fail gates: the general failure-criteria recipe.
• LLM performance and load testing: run the baseline this page compares against.
• AI API scalability: stress, spike, and soak: apply failure criteria to stress and soak runs to detect saturation automatically.
• Streaming and token latency testing: emit llm_ttft_ms for precise TTFT criteria.