AI load testing — do and don't
The following do/don’t pairs consolidate practical wisdom from running load tests against LLM inference endpoints, RAG pipelines, and streaming AI APIs. Each pair is grounded in a real failure mode. Cross-links point to the relevant deep-dive page.
Latency measurement
Section titled “Latency measurement”Do: measure TTFT and end-to-end latency as separate metrics
For streaming endpoints, time to first token (TTFT) and total generation time are fundamentally different user-experience signals. TTFT tells you when the user first sees a response; total latency tells you when they can act on it. Collapsing them into a single http_req_duration metric hides TTFT regressions behind a long generation tail.
Use a k6 custom Trend metric (llm_ttft_ms) to capture TTFT separately from http_req_duration. Set independent failure criteria thresholds for each.
See Streaming and token latency testing.
Don’t: treat non-streaming latency as a proxy for streaming UX
A non-streaming call waits for the full response before returning. Users of streaming UIs experience TTFT (typically < 1 s) not total latency (typically 5–20 s). A non-streaming load test can show a healthy p95 of 4 s while the streaming TTFT p95 is 2.5 s — which is unacceptable for a chat UI. Test the streaming path with streaming enabled.
Do: set an explicit HTTP timeout of 30–60 seconds for LLM requests
The k6 default timeout is 10 seconds. An LLM generating 512 tokens at 30 tokens/s takes ~17 s. Without an explicit timeout, long-running requests are incorrectly classified as errors. Set timeout: '60s' in your k6 options or the equivalent in Taurus.
Prompt and output design
Section titled “Prompt and output design”Do: fix max_tokens in every load-test run
Output token count is the primary driver of LLM response time. If max_tokens varies across requests (or is left as the model default), your p95 latency is measuring a mix of short and long generations — not the behaviour of your actual workload. Fix max_tokens to the value that represents your production use case, and document it alongside the run.
Don’t: use a single repeated prompt for all virtual users
A single prompt triggers the same inference path and, in many inference servers, benefits from KV-cache hits. This produces an artificially fast result that does not reflect production traffic. Use a CSV of realistic prompts — at least 50–100 distinct inputs for a basic test, 200+ for a RAG test to avoid vector-DB cache bias.
See RAG pipeline load testing.
Do: model realistic prompt sizes
A 20-token prompt and a 1 000-token RAG-context prompt hit the same endpoint but produce very different input token costs and TTFT values. Run your test at the prompt size that matches your production workload. If your app uses long context windows (e.g., injecting retrieved documents), test with representative context lengths — not the minimal happy-path prompt.
See Inference cost and token budget testing.
Concurrency and ramp-up
Section titled “Concurrency and ramp-up”Do: start with conservative concurrency (5–10 VUs) and ramp slowly
LLM inference is GPU-bound. A 20-GPU node that handles 30 VUs gracefully may saturate at 35 VUs with a sharp latency inflection and error burst. Start at low concurrency, observe steady-state metrics, then increase VUs in steps. A 2-minute ramp-up for every 20 additional VUs gives GPU workers time to settle.
Don’t: jump straight to 100 VUs before knowing the saturation point
Starting a test at 100 VUs without a baseline produces a single data point: “the system is saturated.” You learn nothing about the saturation knee — the maximum concurrency you can sustain before latency degrades. Discover the knee first with a stepping stress test, then set production concurrency below it.
See AI API scalability — stress, spike, and soak.
Do: right-size concurrency to your production workload
Concurrency in a load test is not the same as user count. For LLM endpoints where each request takes 2–5 s, 20 concurrent users translates to 20 in-flight requests simultaneously — far fewer than 20 interactive users who spend most of their time reading. Map your expected simultaneous-inference demand, not your active-user count, to VU count.
Streaming and SSE
Section titled “Streaming and SSE”Do: test streaming at the cadence your app expects
If your application streams tokens to a chat UI, run your load test with "stream": true and parse SSE events. A non-streaming test validates batch throughput but not the streaming path. Streaming adds HTTP connection hold-open cost that non-streaming tests do not reveal.
Don’t: use the default k6 10 s timeout for streaming responses
Streaming a 512-token response at 30 tokens/s takes ~17 s. Set timeout: '60s' (or longer for large max_tokens values). Timeouts firing mid-stream appear as errors in your results, inflating the error rate and producing false negatives in failure criteria checks.
Cost accounting
Section titled “Cost accounting”Do: track token consumption in every load-test run
Latency data without token counts is an incomplete result for AI endpoints. A run that improved p95 latency by 20% by reducing average output tokens by 40% looks like an improvement — but may indicate the model is truncating outputs. Always record prompt_tokens, completion_tokens, and total from the usage field and emit them as custom k6 metrics.
Do: project monthly cost from your load profile before launch
Multiply (requests/sec) × (seconds/month) × (cost/request) to get monthly spend. Do this before you launch. LLM inference costs compound with traffic in a way that is easy to underestimate: 8 RPS at $0.00035/request = $7,200/month. Run the calculation from your load-test token data.
See Inference cost and token budget testing.
Don’t: treat 429 responses as free
When a hosted LLM API rate-limits you (HTTP 429), the request failed — but if your application auto-retries, those retries consume quota and cost. Under load, a rate-limit cascade can result in actual cost exceeding your model estimate because every failed request generates additional retry requests. Track 429 rate as a separate failure criterion.
Rate limits and 429 handling
Section titled “Rate limits and 429 handling”Do: watch 429 rate as a distinct failure criterion
429 rate-limit errors require a different response than 503 server errors. A cluster of 429s indicates quota exhaustion — the fix is to increase your rate-limit tier or reduce concurrency. A cluster of 503s indicates the inference server is overwhelmed — the fix is to scale capacity. Conflating them into a single “error rate” metric obscures the cause.
See AI load test failure criteria.
Don’t: run high-concurrency load tests against production LLM APIs without quota headroom
A 100-VU load test against an LLM API with a 60-RPM quota will immediately saturate the quota. Check your provider’s TPM/RPM limits before running. If you need to test at production-level concurrency, request a quota increase or use a staging API key with a higher limit — don’t discover the limit by hammering production.
Warm vs cold model state
Section titled “Warm vs cold model state”Do: account for model warm-up in your ramp-up design
Many inference servers — especially self-hosted ones using vLLM, TGI, or Ollama — have GPU workers that initialize KV-cache structures on first use. The first 30–60 seconds of a test with a new model loaded can show elevated TTFT that drops as the worker pool warms up. Use a 2-minute minimum ramp-up and evaluate steady-state metrics from the plateau, not the ramp.
Do: test the cold-start path deliberately with a spike test
If your system autoscales on GPU demand, test what happens when traffic spikes from zero to production concurrency. Autoscaling an inference cluster takes 3–10 minutes. During that window, users experience 503 errors or 20–30 s TTFT as the new GPU worker initialises. A spike test quantifies this cold-start window so you can decide on keep-warm strategies.
See AI API scalability — stress, spike, and soak.
Failure criteria and CI
Section titled “Failure criteria and CI”Do: gate every AI load run with explicit failure criteria
A run that ends with status “Finished” is not a passing run — it means the test completed, nothing more. Encode your SLOs into failure criteria: p95 TTFT > 800 ms → fail, error rate > 2% → fail, throughput < 3 RPS → fail. This produces an automatic Passed / Failed verdict that CI pipelines can consume without human review.
Don’t: set criteria tighter than your measured baseline on the first run
If you set p95 latency > 2 000 ms → fail before you have ever measured the model’s actual p95, you will either always pass (criteria too loose) or always fail (criteria too tight). Run three to five uncriteria’d baseline runs first to learn the normal range, then set thresholds at 120–150% of baseline p95.
See AI load test failure criteria.
Do: compare runs after every model upgrade, prompt change, or infrastructure change
Use MaxoPerf’s run comparison feature to get a quantitative Δ between the candidate and the baseline. A model upgrade that improves quality but adds 40% to p95 TTFT may not be acceptable for your latency SLO. Proactive comparison catches regressions that a threshold-only gate misses when the regression is gradual.
Where to go next
Section titled “Where to go next”- AI testing glossary — definitions for every metric mentioned above
- LLM performance and load testing — MaxoPerf perf testing for LLM endpoints, end to end
- AI load test failure criteria — wiring failure criteria into CI
- How to use this library — page conventions and callout types