AI API scalability — stress, spike, and soak

LLM and inference APIs fail differently from conventional REST services. GPU memory is finite, worker pools are small, and autoscaling lag can be minutes rather than seconds. A standard stress test exposes these failure modes; a spike test reveals cold-start and autoscaling behaviour; a soak test catches KV-cache memory leaks and slow context-accumulation bugs. This page maps each test type to the AI-specific failure mode it targets.

Before you start

• Read LLM performance and load testing — run a baseline load test first so you know steady-state p95 latency before stressing.
• Read Stress test, Spike test, and Soak / endurance test for the general test-type framing.
• Run all stress / spike / soak tests against staging, never production. Inference servers under stress can OOM-crash and affect other tenants.

AI-specific failure modes

Failure mode	Symptoms	Test type that reveals it
GPU saturation	p95 latency climbs linearly with VUs; throughput plateaus	Stress
KV-cache exhaustion	Sudden spike in 500 errors with out of memory body	Stress (high max_tokens)
Worker pool saturation	Queue depth grows; latency climbs but RPS stays flat	Stress
Cold-start lag	First requests in a spike take 5–30 s while GPU worker initialises	Spike
Autoscaling lag	Errors spike during the scale-out window before new instances are ready	Spike
Rate-limit cascade	429 errors cluster at the start of a spike	Spike
Memory leak (KV cache)	p95 latency drifts upward over hours; TPOT grows slowly	Soak
Connection pool exhaustion	503 errors appear only after several hours of sustained load	Soak

Stress test: finding the GPU saturation knee

A stress test ramps VUs steadily past the baseline load to find where latency inflects and errors begin.

Taurus YAML — stress profile

execution:
  - concurrency: 80          # target: 4× baseline load (baseline was 20 VUs)
    ramp-up: 15m             # slow ramp — 1 VU every ~11 s to observe each step
    hold-for: 5m             # brief hold at peak
    scenario: llm-stress

scenarios:
  llm-stress:
    requests:
      - label: chat-completions
        url: https://inference.example.com/v1/chat/completions
        method: POST
        headers:
          Content-Type: application/json
          Authorization: "Bearer ${LLM_API_KEY}"
        body: >
          {
            "model": "llama-3-8b-instruct",
            "messages": [{"role": "user", "content": "Write a haiku about performance testing."}],
            "max_tokens": 64,
            "stream": false
          }

Note

Use a short max_tokens value (32–128) for stress tests. Longer outputs mean each request holds a GPU context longer, reducing effective parallelism. A stress test with max_tokens: 2048 will saturate a 4-GPU node at just 6–8 VUs — too few data points to see the saturation curve clearly.

Console walk-through

1. Duplicate your baseline load test. Rename it llm-stress-80vu. Update the Taurus YAML with concurrency: 80 and ramp-up: 15m.

2. In the Configuration tab set Virtual users to 80, Ramp-up to 15m, Duration to 20m.

3. Add a failure criterion: Error rate > 5% → fail — this captures the breaking point automatically.

4. Click Run. Watch the Overview tab live — the latency chart should show a clear inflection where p95 begins a sustained climb.

5. When the run ends, note the VU count at the inflection. That is your GPU saturation point for this model and hardware configuration.

Reading the stress result

• Latency inflection — the VU count where p95 transitions from flat to a climbing slope. This is the saturation knee: the maximum VUs you can sustain without latency degradation.
• Error rate step-change — 429s appear first (rate-limit quota), then 503/504s (inference server overwhelmed). A 503 cluster at a specific VU count indicates the worker pool ceiling.
• Throughput plateau — RPS stops climbing even as VUs increase. This is the inference server’s maximum token-generation throughput. Above this point, adding VUs only increases queue depth.

Spike test: cold-start and autoscaling lag

An inference server that autoscales on GPU demand can take 3–10 minutes to provision and warm a new GPU instance. A spike test with a near-instant VU surge reveals the cold-start window.

k6 spike script

import http from 'k6/http';
import { check } from 'k6';
import { Rate } from 'k6/metrics';

const coldStartErrors = new Rate('cold_start_error_rate');

export const options = {
  stages: [
    { duration: '2m',  target: 5  },   // low baseline (warm state)
    { duration: '30s', target: 60 },   // near-instant surge (spike)
    { duration: '3m',  target: 60 },   // hold at spike — autoscaling window
    { duration: '30s', target: 5  },   // drop back
    { duration: '2m',  target: 5  },   // recovery observation
  ],
  thresholds: {
    http_req_duration:      ['p(95)<10000'],
    cold_start_error_rate:  ['rate<0.10'],  // tolerate up to 10% errors during cold-start window
  },
};

const ENDPOINT = 'https://inference.example.com/v1/chat/completions';
const API_KEY  = __ENV.LLM_API_KEY;

const PAYLOAD = JSON.stringify({
  model: 'llama-3-8b-instruct',
  messages: [{ role: 'user', content: 'What is 2 + 2?' }],
  max_tokens: 32,
  stream: false,
});

export default function () {
  const res = http.post(ENDPOINT, PAYLOAD, {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${API_KEY}`,
    },
    timeout: '30s',
  });

  const failed = res.status !== 200;
  coldStartErrors.add(failed);

  check(res, { 'status 200': (r) => r.status === 200 });
}

The cold_start_error_rate metric lets you distinguish the transient cold-start error window from sustained errors after autoscaling should have completed. If cold_start_error_rate is high throughout the 3-minute hold phase, autoscaling either hasn’t triggered or hasn’t finished provisioning.

Soak test: KV-cache memory leak and connection exhaustion

A soak test holds steady load for 4–8 hours to detect slow-growing resource leaks.

What AI soak tests reveal that short tests miss

• KV-cache accumulation: some inference servers improperly evict KV caches, causing memory to grow with each request. p95 TTFT drifts upward as available GPU memory shrinks over hours.
• TPOT drift: tokens/sec degrades slowly as the model weights compete with growing cached states for GPU VRAM bandwidth.
• Connection pool exhaustion: HTTP connection pools to the vector DB or embedding service are not properly recycled and exhaust over 4–6 hours.

Taurus YAML — soak profile

execution:
  - concurrency: 20          # same as baseline load test
    ramp-up: 3m
    hold-for: 6h             # 6-hour soak
    scenario: llm-soak

scenarios:
  llm-soak:
    requests:
      - label: chat-completions
        url: https://inference.example.com/v1/chat/completions
        method: POST
        headers:
          Content-Type: application/json
          Authorization: "Bearer ${LLM_API_KEY}"
        body: >
          {
            "model": "llama-3-8b-instruct",
            "messages": [{"role": "user", "content": "Describe a load testing best practice."}],
            "max_tokens": 256,
            "stream": false
          }

Tip

Use MaxoPerf’s scheduled runs to start the soak at an off-peak time (e.g. 22:00) so results are ready to review the next morning. See Schedule recurring tests.

Reading the AI soak result

Watch for drift rather than peaks:

• p95 latency trend — flat for 6 hours is the target. Any upward slope (even 100 ms per hour) indicates a resource leak.
• Throughput decline — if RPS drops at constant VUs, the server is slowing down. Correlate with inference server heap/VRAM metrics.
• Error rate timing — errors appearing only after 3–4 hours (not at run start) are characteristic of connection pool or file-descriptor exhaustion.

Do / don’t

Do	Don’t
Run a baseline load test before any stress / spike / soak	Stress an endpoint before you know its steady-state behaviour
Use short max_tokens for stress tests to see more VU steps before saturation	Use large max_tokens for stress — you saturate at too few VUs for a useful curve
Observe the recovery window in spike tests (2 min minimum after VU drop)	End the spike run at the VU peak — you miss whether the system recovers
Run soak tests on staging with monitoring enabled for VRAM and heap	Run AI soak tests on production shared with real users
Note the exact VU count at the saturation knee for capacity planning	Report only peak VU count — the knee is the actionable metric

Where to go next

• Stress test — general stress test framing and console walk-through.
• Spike test — spike test framing: recovery window, autoscaling, circuit breakers.
• Soak / endurance test — soak framing: drift, scheduling, long-run interpretation.
• AI load test failure criteria — automate the pass/fail verdict for AI stress runs.
• LLM performance and load testing — run the baseline first.
• Streaming and token latency testing — measure TTFT drift during the soak hold phase.