LLM performance and load testing

Load-testing a language model endpoint is fundamentally an HTTP load test — but the metrics that matter are different from a conventional REST API. This page explains how to run a structured LLM load test in MaxoPerf, which AI-specific metrics to collect, and how to interpret what you see in the run-detail tabs.

Before you start

• Read HTTP and REST load testing — the request mechanics are identical.
• Read Taurus fundamentals or k6 scripts on Maxoperf depending on your preferred engine.
• You need an LLM endpoint that accepts HTTP POST requests with a JSON body (OpenAI-compatible chat-completions format or a custom inference API). The endpoint does not need to be a commercial API — self-hosted inference servers (vLLM, Ollama, TGI) work the same way.

AI-specific metrics

Standard load tests measure response time and throughput. LLM endpoints add a layer of token-aware metrics:

Metric	Definition	Why it matters
TTFT (time to first token)	Latency from request sent to first byte of the response body received	Governs perceived responsiveness in streaming UIs
TPOT (time per output token)	Time between successive tokens in a streaming response	Affects reading pace for streamed output
End-to-end latency	Time from request sent to full response received	Total user wait for non-streaming calls
Throughput (RPS)	Requests completed per second across all VUs	Capacity of the inference stack
Tokens/sec (output)	Output tokens generated per second across all VUs	GPU/inference throughput; correlates with cost
Concurrency	Simultaneous in-flight requests	Maps to number of parallel GPU contexts
p95 / p99 latency	95th / 99th percentile of end-to-end latency	Tail behaviour under real concurrent load
Error rate	Fraction of non-2xx responses	Includes 429 rate-limit, 503 overload, 504 timeout
429 rate	Fraction of rate-limit rejections specifically	Reveals provisioned quota vs demand gap
Cost per request	Token count × price per token for the model	Enables load-profile-based monthly cost projection

Note

MaxoPerf measures HTTP response timing — not model-internal metrics. TTFT is approximated by time to first byte (TTFB) on a streaming response. For non-streaming calls, TTFB equals end-to-end latency. True per-token timing requires streaming (see Streaming and token latency testing).

A real Taurus YAML for an LLM endpoint

execution:
  - concurrency: 20          # 20 simultaneous inference requests
    ramp-up: 2m              # gentle ramp — GPU workers need warm-up time
    hold-for: 5m             # sustained plateau to capture steady-state metrics
    scenario: llm-chat

scenarios:
  llm-chat:
    requests:
      - label: chat-completions
        url: https://inference.example.com/v1/chat/completions
        method: POST
        headers:
          Content-Type: application/json
          Authorization: "Bearer ${LLM_API_KEY}"   # injected via MaxoPerf secrets
        body: >
          {
            "model": "llama-3-8b-instruct",
            "messages": [
              {"role": "user", "content": "Summarise the benefits of load testing in three sentences."}
            ],
            "max_tokens": 256,
            "temperature": 0.7,
            "stream": false
          }
        assert:
          - contains:
              subject: http-code
              value: '200'
          - contains:
              subject: body
              value: '"choices"'

Key points about this YAML:

• concurrency: 20 — models 20 parallel users making inference requests simultaneously. Start conservatively; LLM inference is GPU-bound and saturates quickly.
• ramp-up: 2m — a slow ramp avoids overwhelming the KV cache warm-up and lets autoscaling respond if it is enabled.
• max_tokens: 256 — fix the output budget so response times are comparable across runs. Variable output length makes latency comparisons unreliable.
• stream: false — for this first test, get full responses. See Streaming and token latency testing for SSE variants.
• The Authorization header references a MaxoPerf secret — never hard-code API keys. See Manage test secrets.

A real k6 script for an LLM endpoint

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 20 },   // ramp to 20 VUs
    { duration: '5m', target: 20 },   // hold plateau
    { duration: '1m', target: 0 },    // ramp down
  ],
  thresholds: {
    // p95 end-to-end latency under 3 s (adjust for your model/hardware)
    http_req_duration: ['p(95)<3000'],
    // error rate under 2%
    http_req_failed: ['rate<0.02'],
  },
};

const LLM_ENDPOINT = 'https://inference.example.com/v1/chat/completions';
const API_KEY = __ENV.LLM_API_KEY;  // injected from MaxoPerf secrets

const PAYLOAD = JSON.stringify({
  model: 'llama-3-8b-instruct',
  messages: [
    { role: 'user', content: 'Summarise the benefits of load testing in three sentences.' },
  ],
  max_tokens: 256,
  temperature: 0.7,
  stream: false,
});

export default function () {
  const params = {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${API_KEY}`,
    },
    timeout: '30s',   // LLM inference can be slow; set an explicit timeout
  };

  const res = http.post(LLM_ENDPOINT, PAYLOAD, params);

  check(res, {
    'status 200':          (r) => r.status === 200,
    'has choices field':   (r) => r.json('choices') !== undefined,
    'not rate-limited':    (r) => r.status !== 429,
  });

  // No sleep — LLM inference already has long natural think time
}

How to create and run the test in MaxoPerf

1. In the MaxoPerf console, go to Tests → New test. Give the test a name such as llm-chat-load-20vu.

2. Open the Files tab. Upload your Taurus YAML (or k6 .js file) and mark it as the Entrypoint. MaxoPerf auto-detects the engine.

3. Open Settings → Secrets and add LLM_API_KEY with your inference API key. This secret is injected as ${LLM_API_KEY} at run time — it never appears in the uploaded file.

4. Open the Configuration tab. Set Virtual users to 20, Ramp-up to 2m, and Duration to 8m (2 m ramp + 5 m hold + 1 m ramp-down). Select your runner location closest to the inference server to minimise network overhead in measurements.

5. In the Failure criteria section, add: p95 latency > 3000 ms → fail and Error rate > 2% → fail.

6. Click Run. The run detail page opens automatically.

How to read the results

Overview tab

• Throughput (RPS) — for an LLM endpoint with max_tokens: 256, expect 1–5 RPS per VU depending on the model and hardware. A plateau that never reaches the expected RPS suggests the model is GPU-saturated.
• p95 / p99 latency — LLM latency is orders of magnitude higher than a REST API. A p95 of 2–5 seconds is normal for a 7B–13B parameter model at moderate concurrency. Watch for the latency curve climbing without plateau — that means the queue is growing faster than the model drains it.
• Error rate — 429s appear when the endpoint’s rate-limit quota is exceeded. 503/504s appear when the inference server is overloaded or out-of-memory.

Errors tab

• A cluster of 429 errors early in the run indicates a provisioned quota problem (not a performance problem). Contact your LLM API provider or adjust your rate limits.
• 504 timeouts under load indicate the inference server is saturating — either reduce concurrency or scale up GPU capacity.
• 500 errors with out of memory in the body indicate the KV cache is exhausted — reduce max_tokens or add GPU nodes.

Do / don’t

Do	Don’t
Fix max_tokens so response lengths (and timings) are comparable across runs	Let max_tokens be open-ended — variable output length makes p95 meaningless
Use Authorization via MaxoPerf secrets — never hard-code API keys	Put API keys in uploaded test files
Start with 5–10 VUs and increase — LLM endpoints saturate fast	Jump to 100 VUs before knowing the saturation point
Label requests clearly (label: chat-completions) for per-endpoint breakdown	Use unlabelled requests — the results panel shows unnamed instead of meaningful names
Set an explicit HTTP timeout (30–60 s) in k6	Rely on the default 10 s timeout — inference often takes longer
Run against staging / non-production inference endpoints	Load-test production LLM APIs without provider approval and a rate-limit increase

Where to go next

• Streaming and token latency testing — measure TTFT precisely with SSE/chunked responses.
• Inference cost and token budget testing — project monthly cost from this load profile.
• RAG pipeline load testing — load-test retrieval + generation as one end-to-end endpoint.
• AI API scalability: stress, spike, and soak — push the endpoint past saturation.
• AI load test failure criteria — automate pass/fail gating on p95 TTFT and error rate.
• Stress test — general stress-test framing to apply here.
• Cookbook: Failure criteria pass/fail gates — configuring thresholds in the console.