AI performance testing glossary
This glossary defines the performance and load-testing terminology used throughout the AI Testing section. Each term includes a teaching definition and a note on how it appears in MaxoPerf workflows or results. For one-card canonical definitions, see the SEO Glossary.
Term index
Section titled “Term index”B · C · Co · Co · Co · E · G · I · K · L · P · Q · R · S · T · Ti · Ti · To · To
Time to first token (TTFT)
Section titled “Time to first token (TTFT)”The elapsed time from sending an inference request to receiving the first token of the response. For streaming LLM APIs, this is the latency to the first data: chunk in the SSE stream. TTFT is driven by the prefill phase — the model processing all input tokens before generating any output.
In MaxoPerf: TTFT is measured as the latency to the first SSE event in a streaming response. In k6, emit TTFT as a custom Trend metric (llm_ttft_ms) by recording Date.now() at the first parsed data: line. Add a separate failure criterion on llm_ttft_ms p95 — for example, > 800 ms → fail for a chat UI.
See Streaming and token latency testing.
TPOT (time per output token)
Section titled “TPOT (time per output token)”The average time to generate each output token after the first. TPOT is driven by the decode phase — the model auto-regressively generating one token at a time from its previously generated context. TPOT determines the streaming “reading pace”: high TPOT causes visible stutter in a chat UI.
In MaxoPerf: TPOT = (total generation time − TTFT) / (output tokens − 1). You cannot measure TPOT directly from HTTP metrics; approximate it by recording the time from first to last SSE chunk and dividing by output token count.
Inter-token latency (ITL)
Section titled “Inter-token latency (ITL)”The time between the arrival of successive tokens in a streaming response. ITL and TPOT measure the same concept from different perspectives: ITL is per-inter-token interval (observed at the client); TPOT is the average per-output-token rate (computed post-hoc). In practice, inference literature uses the terms interchangeably.
Time to last token (TTLT)
Section titled “Time to last token (TTLT)”The elapsed time from request sent to the final token of the response received — the full streaming latency. For non-streaming calls, TTLT equals end-to-end latency. TTLT = TTFT + (output_tokens × TPOT).
In MaxoPerf: http_req_duration measures TTLT (last byte received). For streaming endpoints, this is the full generation time, not the user-visible response start.
End-to-end latency
Section titled “End-to-end latency”For non-streaming LLM calls, the total time from request sent to full response body received — equivalent to TTLT. For streaming calls, end-to-end latency is TTLT, while the user-visible response begins at TTFT.
In MaxoPerf: The standard http_req_duration metric. Set separate SLO thresholds for TTFT and end-to-end latency — they answer different questions.
Tokens per second (tokens/sec)
Section titled “Tokens per second (tokens/sec)”The rate at which the inference system generates output tokens. Measured either per-request (generation speed) or across all concurrent requests (system throughput). Higher tokens/sec indicates more efficient GPU utilisation.
In MaxoPerf: Estimate aggregate tokens/sec by emitting a llm_output_tokens counter in your k6 script and dividing the total by test duration in seconds. A declining tokens/sec under increasing concurrency indicates the inference backend is saturating.
Throughput (RPS)
Section titled “Throughput (RPS)”Requests completed per second across all virtual users. For LLM endpoints with fixed max_tokens, RPS is the primary capacity metric — it tells you how many inferences the system can sustain per unit time.
In MaxoPerf: The Overview tab shows throughput in RPS over time. During a load test, throughput should plateau at a sustainable level during the hold phase. A throughput plateau that coincides with climbing latency signals that the system is at its processing ceiling.
Latency percentiles (p50, p95, p99)
Section titled “Latency percentiles (p50, p95, p99)”Statistical descriptions of the latency distribution across all requests in a run. p95 means 95% of requests complete within that time; p99 means 99%. Mean latency is misleading for LLM endpoints because variable output lengths produce heavy-tailed distributions where the mean is pulled up by long-tail outliers.
In MaxoPerf: Set failure criteria on p95 (not mean) as your primary SLO metric. p99 widens relative to p95 when there are outlier long-generation requests — a high p99/p95 ratio indicates a wide distribution of output lengths or occasional GPU scheduling jitter.
Concurrency / virtual users
Section titled “Concurrency / virtual users”The number of simultaneous in-flight requests. In MaxoPerf (and k6/Taurus), this is the VU count. For LLM endpoints, each concurrent VU holds one open GPU context during generation. At high concurrency, GPU memory and worker pool limits are exhausted before network bandwidth.
In MaxoPerf: Start conservative (5–10 VUs) and increase incrementally. The VU count at which p95 latency begins a sustained climb is the saturation knee — the maximum concurrency you can sustain without degradation. See AI API scalability — stress, spike, and soak.
Queueing and saturation
Section titled “Queueing and saturation”When incoming requests arrive faster than the inference server can process them, they queue. Queuing shows up as rising p95 latency at constant or declining throughput — the system is accepting requests but not completing them faster. Queueing is the primary signal that a system has crossed its saturation point.
In MaxoPerf: Watch for the pattern where VU count climbs but RPS plateaus: that plateau is the saturation ceiling. Latency climbing while RPS is flat means every additional VU is adding to queue depth, not adding throughput.
GPU saturation
Section titled “GPU saturation”The state where the GPU’s memory bandwidth, VRAM, or compute is fully utilised and the inference server cannot process additional requests any faster. GPU saturation produces a characteristic pattern: throughput plateaus, p95 latency begins climbing linearly, and error rate increases as KV-cache or worker pool limits are exceeded.
In MaxoPerf: Identify the saturation knee in stress-test results — the VU count where the latency chart transitions from flat to climbing. That is the maximum supported concurrency for this model and hardware configuration. See AI API scalability — stress, spike, and soak.
KV cache
Section titled “KV cache”The key-value cache that an inference server maintains for each in-flight request, storing intermediate attention computations across the input tokens. The KV cache is the primary consumer of GPU VRAM during inference. Each active request holds a KV cache proportional to its context length; when VRAM is exhausted, new requests are rejected or degraded.
In MaxoPerf: KV-cache exhaustion appears as sudden 500 errors with out of memory bodies. Reduce max_tokens or the number of concurrent VUs. In soak tests, a slow KV-cache memory leak appears as upward drift in p95 TTFT over several hours.
Cold start / warm start
Section titled “Cold start / warm start”A cold start occurs when an inference server must load model weights and initialise GPU workers before handling the first request — adding 10–60 seconds of latency to the first requests after startup or after a scale-out event. A warm start is when GPU workers and model weights are already resident in VRAM.
In MaxoPerf: Cold-start lag appears as elevated TTFT in the first 30–60 seconds of a test or in the first window of a spike test. Use a ramp-up phase to let workers warm before evaluating steady-state metrics. Test the cold-start path deliberately with a spike test to quantify autoscaling lag.
Rate limit / HTTP 429
Section titled “Rate limit / HTTP 429”A rate-limit response (HTTP 429) from a hosted LLM API indicating that the client has exceeded the provisioned token-per-minute (TPM) or request-per-minute (RPM) quota. 429 responses are not inference-server errors — they are quota-enforcement responses and require a different remediation path (quota increase, request shaping) than 503/504 server errors.
In MaxoPerf: Track 429 rate as a separate custom metric or failure criterion, independent of the overall error rate. A cluster of 429s early in a ramp-up indicates quota is insufficient for the target concurrency. See AI load test failure criteria.
Prompt tokens / output tokens
Section titled “Prompt tokens / output tokens”The two billing buckets for hosted LLM APIs. Prompt tokens are the input tokens (system message + user message + any injected context). Output tokens are the model-generated response tokens. Most providers price output tokens 2–10× higher than input tokens.
In MaxoPerf: Extract both from the usage field in the response body and emit them as k6 Counter metrics. Use the totals to compute per-request cost and project monthly spend from the load profile. See Inference cost and token budget testing.
Cost per token / cost per request
Section titled “Cost per token / cost per request”The inference cost attributed to a single token or a single complete request. For hosted APIs: cost = (prompt_tokens / 1000) × input_price + (output_tokens / 1000) × output_price. For self-hosted models: cost is compute time × hourly server cost / requests completed.
In MaxoPerf: Emit a k6 Trend metric (llm_cost_usd_per_req) computed from the usage field. The p50 and p95 of this metric, combined with RPS, give you the cost model for your load profile. See Inference cost and token budget testing.
Context window (as a latency factor)
Section titled “Context window (as a latency factor)”The maximum number of tokens an LLM can process in a single inference call — input + output combined. For load testing, context-window-filling requests (long system prompts + many retrieved RAG documents + long output) are the worst case for TTFT, TPOT, and cost. The KV cache for a 128 K-token context is 64× larger than for a 2 K-token context.
In MaxoPerf: Test at the prompt lengths your application actually uses. A short-prompt test that shows p95 = 800 ms does not tell you what happens when your RAG system injects 4 000 tokens of retrieved context. Run separate load tests at representative context lengths.
Server-sent events (SSE) / chunked transfer
Section titled “Server-sent events (SSE) / chunked transfer”Server-sent events (SSE) is the HTTP streaming mechanism most LLM APIs use to deliver tokens progressively. The server sends Content-Type: text/event-stream and emits data: {...} lines as tokens are generated. Chunked transfer encoding is the underlying HTTP/1.1 mechanism that delivers body bytes without a predetermined Content-Length.
In MaxoPerf: k6 buffers the full SSE response before the script function returns. Parse SSE chunks from res.body.split('\n') and record timestamps to extract TTFT. See Streaming and token latency testing.
Batching
Section titled “Batching”The practice of grouping multiple inference requests into a single GPU pass to improve throughput. Continuous batching (dynamic batching) is the technique used by modern inference servers (vLLM, TGI) to pack requests of different lengths into the same GPU kernel without waiting for all to complete before starting new ones.
In practice: Batching improves aggregate tokens/sec but increases per-request TTFT (because a request may wait for a batch slot). Under load testing, the impact of batching is visible as a TTFT floor that rises slightly as concurrency increases — each request waits for a batch slot before its prefill begins.
Streaming throughput
Section titled “Streaming throughput”The aggregate rate at which the inference system delivers output tokens to all concurrent clients, measured in tokens/sec across all VUs. Streaming throughput is the GPU-level metric that determines whether the system can keep up with concurrent streaming demand — if streaming throughput falls below (VUs × expected_tokens/sec_per_user), clients experience stutter.
In MaxoPerf: Compute streaming throughput as total_output_tokens / test_duration_seconds. A declining streaming throughput at constant VU count (during a soak test) indicates VRAM or bandwidth degradation.
Where to go next
Section titled “Where to go next”- AI testing do and don’t — best-practice pairs using these concepts
- LLM performance and load testing — run your first LLM load test in MaxoPerf
- Streaming and token latency testing — measure TTFT and TPOT in practice
- Inference cost and token budget testing — cost metrics in depth
- Academy Glossary — general performance and load testing terms