AI load test failure criteria and run comparison

A MaxoPerf run that ends with status Finished is not automatically a passing run — it just means the test completed. For LLM and inference endpoints, “finished” without criteria means you must manually inspect every chart to decide whether the endpoint met its SLOs. Failure criteria encode your AI SLOs directly into the test, producing an automatic Passed / Failed verdict that CI pipelines, scheduled nightly runs, and model-upgrade comparisons can consume without human review.

Before you start

• Read Cookbook: Failure criteria pass/fail gates — this page builds on that general recipe with AI-specific thresholds.
• You have at least one completed baseline run for the LLM endpoint (from LLM performance and load testing).
• You know your SLO targets: p95 TTFT budget, acceptable error rate, minimum throughput floor.

AI-specific failure criteria

Standard load tests gate on p95 latency and error rate. AI endpoints benefit from additional thresholds:

Criterion	Metric	Typical threshold	Why
TTFT budget	http_req_duration p95 (or llm_ttft_ms custom)	< 800 ms–3 000 ms	Governs UX for chat/copilot features
End-to-end latency	http_req_duration p99	< 10 000 ms	Tail latency — impacts long-tail user experience
Error rate	Overall error rate	< 2–5 %	Includes 429 rate-limit and 503/504 overload
429 rate-limit rate	Custom rate_limit_rate	< 1 %	Flags quota exhaustion independently of server errors
Throughput floor	Throughput (RPS)	> N RPS	Catches runs where the test itself under-loaded (startup errors)
Custom token metric	llm_ttft_ms p95 (from streaming k6 script)	< 800 ms	TTFT when measured precisely via SSE parsing

Step by step: configuring AI failure criteria

1. Open the test in the MaxoPerf console and go to the Configuration tab.

2. Scroll to Failure criteria and click Add criterion.

3. Add the TTFT / latency criterion:

   • Metric: p95 latency
   • Label: All labels (or chat-completions if you labelled your request)
   • Operator: greater than
   • Value: 3000 (milliseconds — adjust to your model’s expected p95)

4. Add the error-rate criterion:

   • Metric: Error rate
   • Operator: greater than
   • Value: 2 (percent)

5. Add the throughput floor:

   • Metric: Throughput (RPS)
   • Operator: less than
   • Value: 3 (minimum expected RPS at your configured VU count — prevents false passes from a misconfigured test)

6. Click Save.

Adding custom metric thresholds

If your k6 script emits a custom metric (e.g. llm_ttft_ms from the Streaming and token latency testing script), MaxoPerf surfaces it in the custom metrics section:

1. In the Failure criteria section, click Add criterion.
2. Select metric: Custom metric.
3. Enter the metric name: llm_ttft_ms.
4. Aggregation: p95.
5. Operator: greater than.
6. Value: 800.
7. Click Add.

This criterion fails the run if the TTFT p95 exceeds 800 ms — a direct latency SLO for streaming UIs.

Tip

Start with loose thresholds (e.g. p95 < 5 000 ms, error rate < 5 %) on your first few runs to establish the normal range. Tighten them incrementally as you build confidence in the baseline. AI endpoints are inherently noisier than conventional REST APIs due to variable generation length and GPU scheduling jitter.

Run comparison: model and endpoint versions

When you upgrade a model, change a prompt template, or swap inference providers, comparing the new endpoint against the baseline run reveals regressions and improvements quantitatively.

How to trigger a comparable run

For a valid comparison, both runs must use:

• The same VU count, ramp-up, and hold duration.
• The same prompt body and max_tokens value.
• The same MaxoPerf runner location.

Create a new test named llm-chat-v2 (or duplicate the baseline test and update the endpoint URL / model name). Run it. Then compare.

Using run comparison in MaxoPerf

1. Open the new run (the candidate).
2. Click Compare in the run detail header. A run-selector dialog appears.
3. Select the baseline run from the same test or a different test with the same scenario.
4. The Overview tab switches to comparison mode: dual-line charts for latency and throughput, with Δ chips showing the delta between runs.

Key delta signals for AI endpoint comparisons:

Δ metric	Interpretation
p95 latency Δ positive	Candidate is slower — expected if the new model is larger
p95 latency Δ negative	Candidate is faster — check output quality hasn’t dropped
Throughput Δ positive	Candidate completes more requests/sec — more efficient inference
Error rate Δ positive	Candidate has more errors — could be quota issue or instability
p99 / p95 ratio widens	Candidate has higher tail variance — more scheduling jitter or longer outlier generations

Reporting dashboard for AI runs

MaxoPerf custom dashboards let you arrange the metrics that matter for AI endpoints into a single view — TTFT percentiles, error rate breakdown, and throughput — rather than switching between tabs.

To build an AI-focused dashboard:

1. Open a completed run and click Dashboard → New dashboard.
2. Add a latency percentile panel with p50, p95, p99 lines. Rename it “TTFT / end-to-end latency”.
3. Add a throughput panel and rename it “Inference RPS”.
4. Add an error rate panel segmented by status code (429 vs 5xx).
5. If your script emits llm_output_tokens, add a custom metric panel and set it to llm_output_tokens rate — this is your tokens/sec proxy.
6. Save the dashboard as “AI endpoint — standard”. MaxoPerf reuses it on every subsequent run for this test.

CI/CD integration

Failure criteria make AI load tests CI-compatible. When a run ends with status Failed, the MaxoPerf API returns the failed status and the criterion that was violated:

# Poll for run completion and surface the verdict
curl -s -H "Authorization: Bearer $MAXOPERF_API_KEY" \
  "https://app.maxoperf.com/api/v1/runs/$RUN_ID" \
  | jq '{status: .status, violations: .failure_criteria_violations}'

A CI job that triggers a run and polls this endpoint can exit non-zero when status == "failed" — blocking a model upgrade deployment automatically. See Run a test from CI/CD for the full pipeline integration guide.

Do / don’t

Do	Don’t
Start criteria loose; tighten over several runs as you learn the normal range	Set criteria tighter than your measured baseline — every run will fail
Use a throughput floor to catch misconfigured tests	Rely solely on latency criteria — a test that sent 1 RPS can pass on latency while proving nothing
Compare runs with identical load parameters for a fair model A/B	Compare runs with different VU counts — latency scales non-linearly with concurrency
Document the SLO thresholds in the test description so reviewers understand the pass bar	Leave criteria values undocumented — future reviewers won’t know if 3000 ms was a deliberate choice
Use run comparison after every model upgrade, prompt change, or infrastructure change	Only review comparison after a failure — proactive comparison catches slow regressions

Where to go next

• Cookbook: Failure criteria pass/fail gates — the general failure-criteria recipe.
• LLM performance and load testing — run the baseline this page compares against.
• AI API scalability: stress, spike, and soak — apply failure criteria to stress and soak runs to automate saturation detection.
• Streaming and token latency testing — emit llm_ttft_ms for precision TTFT criteria.