AI inference load testing

Use case

LLM inference APIs have performance characteristics unlike conventional REST services. Latency depends not just on concurrency but on prompt length, output token count, model size, and the batch-scheduling algorithm running on the GPU. A model server that handles 20 concurrent requests at 50 ms TTFT might saturate at 40 concurrent requests with TTFT climbing past 8 seconds — not because of a bug, but because of how transformer attention scales with batch size.

AI inference load testing maps that relationship: concurrency versus TTFT, throughput, and generation speed. The output is a capacity curve, not a single pass/fail number. That curve informs autoscaling thresholds, GPU provisioning decisions, and the latency targets you can realistically offer in your product SLA.

This use case applies to any team operating a model-serving stack: self-hosted vLLM, TGI (Text Generation Inference), Ollama, or a wrapper around a third-party inference API. The questions are the same: how many concurrent requests can we handle before quality degrades? What is the cost per request at our target concurrency? Where is the batch-scheduling sweet spot?

How MaxoPerf supports it

AI inference endpoints are HTTP endpoints. A completion or chat API — whether OpenAI-compatible, Hugging Face, or custom — accepts HTTP requests and returns responses (streaming or synchronous). MaxoPerf’s HTTP executor handles both patterns.

For streaming endpoints (server-sent events), the test script opens the streaming connection, records the timestamp at the first data chunk (TTFT), and records the timestamp at the stream close (total completion time). Labels separate prompt categories, concurrency tiers, or model variants.

The result breakdown gives you TTFT and total completion time per label across the run. By running multiple labeled scenarios at different concurrency levels — 5, 10, 20, 40 concurrent requests — you build the capacity curve that shows where throughput peaks and TTFT degrades.

Private runner locations allow load-testing a self-hosted model server inside your VPC, where the inference API is not internet-accessible. This is the typical topology for self-hosted model serving.

A concrete inference load test

An AI-native company tests their vLLM-backed completions API before every model deployment:

• Endpoint: POST /v1/completions (OpenAI-compatible); streaming SSE response
• Prompt categories: short (< 100 tokens), medium (100–500 tokens), long (500–2000 tokens) — drawn from a sample of real user prompts
• Concurrency sweep: labeled runs at 5, 10, 20, 30, 40 concurrent requests per prompt category (12 label × concurrency combinations)
• Metrics recorded: TTFT (time to first SSE chunk), total completion time, tokens generated, GPU utilisation (via runner-health custom metrics)
• Findings: the capacity curve showed TTFT below 800 ms up to 20 concurrent requests; at 30 requests TTFT for long prompts spiked to 4.1 s due to KV-cache pressure; autoscaling threshold set at 25 concurrent requests per GPU to maintain TTFT < 1 s
• Cost calibration: at 20 concurrent requests, average cost per 1,000 output tokens is $0.0032 on the provisioned GPU tier — within the target product margin

Practical rollout

1. Profile your prompt distribution — before writing the test, analyse your expected or actual prompt and completion length distribution. Group into 2–3 length categories and build a labeled scenario for each.
2. Run a concurrency sweep — execute labeled runs at 5, 10, 20, and 40 concurrent requests (or whatever range covers your expected load). This produces a capacity curve rather than a single data point.
3. Measure TTFT separately from total completion time — both matter but for different reasons. TTFT determines perceived responsiveness; total completion time determines throughput capacity. Optimising for one can hurt the other.
4. Test after every model or framework change — quantisation changes, VRAM upgrades, framework upgrades (vLLM, TGI versions), and model fine-tuning all affect inference performance. Treat model deployments like code deployments and run the capacity sweep before promoting.
5. Build a cost model alongside the performance model — note GPU utilisation and request count per labeled run. Combined with your GPU hourly cost, this gives you cost-per-request at each concurrency level — the data you need for infrastructure sizing decisions.