AI performance testing overview

This section covers performance and load testing of AI/LLM systems using MaxoPerf. An LLM inference endpoint, a RAG pipeline, or an embedding service is, at its HTTP layer, a target you can load-test just like any other API — but the metrics that matter, the failure modes to watch for, and the load-profile decisions you need to make are all fundamentally different from a conventional REST service. This overview explains what those differences are and shows you where to start.

What AI performance testing covers

When you load-test an LLM or inference endpoint, you are measuring how the system behaves under concurrent real-world traffic. The questions this answers are:

• How fast does the model respond? — measured by time to first token (TTFT) for streaming calls and end-to-end latency for non-streaming calls.
• How much can it handle? — measured by requests per second (throughput) and the concurrency level at which latency begins to degrade (the saturation knee).
• How expensive is it? — measured by tokens consumed per request and projected monthly cost from the load profile.
• Does it hold up over time? — measured by soak runs that detect KV-cache memory leaks, TPOT drift, and connection pool exhaustion.
• Where does it break? — measured by stress and spike tests that reveal GPU saturation points, cold-start lag, and rate-limit cascade behaviour.

AI-specific metrics

Standard load tests measure response time and throughput. AI endpoints add a layer of token-aware and inference-aware metrics:

Metric	Definition	Why it matters
TTFT (time to first token)	Latency from request sent to first byte received	Governs perceived responsiveness in streaming UIs
TPOT (time per output token)	Time between successive tokens in a streaming response	Determines reading pace; slow TPOT causes visible stuttering
End-to-end latency	Time from request sent to full response received	Total user wait for non-streaming calls
Throughput (RPS)	Completed requests per second across all VUs	Capacity of the inference stack
Tokens/sec	Output tokens generated per second across all VUs	GPU/inference throughput; correlates with cost
Concurrency	Simultaneous in-flight requests	Maps to number of parallel GPU contexts
p95 / p99 latency	95th / 99th percentile of end-to-end latency	Tail behaviour under real concurrent load
Error rate	Fraction of non-2xx responses	Includes 429 rate-limit, 503 overload, 504 timeout
429 rate	Rate-limit rejections specifically	Reveals provisioned quota vs demand gap
Cost per request	Token count × price per token	Enables load-profile-based monthly cost projection
Time to last token	Elapsed time from first byte to final byte	Total streaming generation time; complements TTFT

Why LLM endpoints are different to load-test

Variable response length. A conventional API returns a fixed-size JSON body. An LLM generates a variable number of output tokens — a request for a one-sentence answer and a request for a 2,000-token essay hit the same endpoint but produce dramatically different latencies. Fix max_tokens in every load test to make runs comparable.

GPU saturation is non-linear. CPU-bound services degrade smoothly under load. GPU-bound inference degrades in steps: performance is flat up to the saturation knee, then latency climbs sharply and errors spike. Finding that knee is the primary goal of AI stress testing.

Cold-start lag. Autoscaling an inference cluster takes 3–10 minutes, not seconds. A spike test for an AI endpoint must hold at peak load long enough to observe whether autoscaling actually rescues the endpoint.

KV-cache effects. A model’s KV cache is GPU memory that grows with request context length. Under sustained load, KV-cache exhaustion produces 500 errors and OOM crashes. Soak tests are required to detect slow KV-cache leaks that a 10-minute run cannot surface.

Streaming changes the latency model. For a non-streaming API, HTTP response time equals total processing time. For a streaming LLM, users see the first token at TTFT (often < 1 s) and then watch tokens arrive at TPOT rate. A non-streaming load test misses the metric your users actually care about.

Rate-limit cascades. Hosted LLM APIs enforce token-per-minute (TPM) and request-per-minute (RPM) quotas. Under load, quota exhaustion produces 429 waves that look like server failures but require different remediation — increasing quota, not scaling servers.

How MaxoPerf fits in

MaxoPerf generates HTTP load against any LLM inference endpoint, chat-completion API, RAG pipeline, or embedding service. You write a Taurus YAML or k6 script that sends realistic prompt payloads; MaxoPerf runs it from cloud runner locations, streams results to the console in real time, and produces latency percentile charts, throughput graphs, and error breakdowns.

For AI endpoints specifically, MaxoPerf supports:

• Custom k6 metrics (e.g. llm_ttft_ms, llm_output_tokens) emitted from your script and surfaced in the results dashboard.
• Failure criteria on p95 TTFT, error rate, throughput floor, and any custom metric — producing an automatic Passed / Failed verdict for CI pipelines.
• Run comparison — side-by-side Δ charts when you upgrade a model, change a prompt, or swap inference providers.
• Multi-region runners — place load generation close to your inference server to isolate network latency from inference latency.
• CSV-parameterised requests — feed a CSV of real user queries to avoid cache bias in vector-database-backed RAG pipelines.
• Scheduled and soak runs — launch 6-hour soak tests at off-peak hours; review drift charts the next morning.

A representative MaxoPerf load profile for an LLM chat endpoint:

execution:
  - concurrency: 20
    ramp-up: 2m
    hold-for: 5m
    scenario: llm-chat

scenarios:
  llm-chat:
    requests:
      - label: chat-completions
        url: https://inference.example.com/v1/chat/completions
        method: POST
        headers:
          Content-Type: application/json
          Authorization: "Bearer ${LLM_API_KEY}"
        body: >
          {
            "model": "llama-3-8b-instruct",
            "messages": [{"role": "user", "content": "Summarise the benefits of load testing."}],
            "max_tokens": 256,
            "temperature": 0.7,
            "stream": false
          }

At 20 VUs with a 2-minute ramp and a 5-minute hold, you get steady-state p95 latency, throughput, and error rate for a baseline. From there you can stress it (ramp to 80 VUs), spike it (0 → 60 VUs in 30 s), or soak it (20 VUs for 6 hours).

Out of scope

This section is about performance: latency, throughput, concurrency, cost, and reliability under load. It does not cover quality evaluation (LLM-as-judge, RAGAS, golden-set regression), safety testing (bias, toxicity, robustness), or security testing (prompt injection, red-teaming). Those disciplines have their own tooling — DeepEval, RAGAS, Garak, and others — and you pair them with MaxoPerf’s performance gate to form a complete pre-promotion pipeline.

Pages in this section

Page	What you learn
LLM performance and load testing	Run a baseline load test on a chat-completions endpoint; read TTFT, RPS, and error charts
Streaming and token latency testing	Measure TTFT precisely from SSE chunks using k6 custom metrics
Inference cost and token budget testing	Extract token counts, compute per-request cost, project monthly spend
RAG pipeline load testing	Load-test retrieval + generation as a single endpoint; avoid cache bias with CSV queries
AI API scalability — stress, spike, and soak	Find the GPU saturation knee, reveal cold-start lag, detect KV-cache memory leaks
AI load test failure criteria	Gate runs on p95 TTFT, error rate, and custom token metrics; compare model versions
AI testing do and don’t	Performance-focused best-practice pairs for LLM load testing
AI testing glossary	Definitions for every AI performance metric with MaxoPerf workflow notes

Where to go next

• LLM performance and load testing — start here; run your first LLM load test in MaxoPerf.
• Test types — stress, spike, soak, and load test types, all applicable to AI endpoints.
• Cookbook — ready-to-run load-test recipes adaptable for LLM and RAG endpoints.
• Failure criteria pass/fail gates — configure automated pass/fail verdicts in the MaxoPerf console.