Streaming and token latency testing

When an LLM API streams its response — using Server-Sent Events (SSE) or chunked transfer encoding — two fundamentally different latency metrics matter: time to first token (TTFT) and time per output token (TPOT). A non-streaming load test captures only end-to-end latency and misses the user experience entirely. This page shows how to instrument streaming requests in MaxoPerf so you can measure both.

Before you start

• Read LLM performance and load testing — this page builds on the baseline load test setup.
• Read k6 scripts on Maxoperf — streaming requires k6 (Taurus native HTTP does not support byte-by-byte SSE parsing).
• Your LLM endpoint must support streaming: set "stream": true in the request body and confirm the response uses Content-Type: text/event-stream or chunked encoding.

Why streaming changes latency interpretation

In a non-streaming call, the client waits for the full response before the connection closes. The HTTP response time is end-to-end latency — total time for the model to generate all max_tokens output tokens.

In a streaming call:

Phase	Latency metric	User experience
Request → first byte of response	TTFT	Time before the user sees anything
First byte → last byte	generation time (≈ output_tokens × TPOT)	Reading pace
Request → last byte	end-to-end latency	Total wait if not streaming

For chat and copilot UIs, TTFT is the critical UX metric — users notice a blank screen for more than ~500 ms. TPOT matters for long generations where the reading pace slows noticeably.

Note

MaxoPerf records HTTP response time (request to last byte). To isolate TTFT you need to parse the first data: chunk timestamp in k6 and emit it as a custom metric — the example below shows exactly how.

k6 script: measuring TTFT from an SSE stream

import http from 'k6/http';
import { check } from 'k6';
import { Trend, Counter } from 'k6/metrics';

// Custom metrics emitted to MaxoPerf results
const ttftTrend = new Trend('llm_ttft_ms', true);        // time to first token, ms
const tokenCounter = new Counter('llm_output_tokens');    // total output tokens seen

export const options = {
  stages: [
    { duration: '2m', target: 15 },   // ramp to 15 VUs
    { duration: '5m', target: 15 },   // hold
    { duration: '1m', target: 0 },    // ramp down
  ],
  thresholds: {
    // End-to-end streaming latency (full response received)
    http_req_duration:  ['p(95)<15000'],   // 15 s for a 512-token streaming response
    // TTFT p95 under 800 ms
    llm_ttft_ms:        ['p(95)<800'],
    // Error rate under 2%
    http_req_failed:    ['rate<0.02'],
  },
};

const ENDPOINT = 'https://inference.example.com/v1/chat/completions';
const API_KEY  = __ENV.LLM_API_KEY;

const PAYLOAD = JSON.stringify({
  model: 'llama-3-8b-instruct',
  messages: [{ role: 'user', content: 'Write a short story about a robot learning to bake.' }],
  max_tokens: 512,
  temperature: 0.7,
  stream: true,   // enable SSE streaming
});

export default function () {
  const startMs = Date.now();
  let firstChunkMs = null;
  let tokenCount = 0;

  const res = http.post(ENDPOINT, PAYLOAD, {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${API_KEY}`,
      'Accept': 'text/event-stream',
    },
    timeout: '60s',
    // k6 streams the body — we read it chunk-by-chunk
    responseType: 'text',
  });

  check(res, { 'status 200': (r) => r.status === 200 });

  // Parse SSE lines from the buffered response body
  if (res.status === 200 && res.body) {
    const lines = res.body.split('\n');
    for (const line of lines) {
      if (line.startsWith('data: ') && line !== 'data: [DONE]') {
        // Record TTFT on the very first data chunk
        if (firstChunkMs === null) {
          firstChunkMs = Date.now() - startMs;
          ttftTrend.add(firstChunkMs);
        }
        // Count tokens (each SSE chunk typically carries 1 token)
        try {
          const chunk = JSON.parse(line.slice(6));
          const delta = chunk?.choices?.[0]?.delta?.content ?? '';
          if (delta) tokenCount++;
        } catch (_) { /* skip malformed chunks */ }
      }
    }
  }

  tokenCounter.add(tokenCount);
}

What this script does

1. Sends a streaming request ("stream": true, Accept: text/event-stream).
2. Parses SSE chunks from the buffered response body (k6 buffers the full stream before the function returns — see the note below on true streaming).
3. Records TTFT as a custom Trend metric (llm_ttft_ms) the moment the first data: line is seen.
4. Counts output tokens to cross-check generation volume across VUs.
5. Applies thresholds — both http_req_duration (full stream) and llm_ttft_ms (TTFT p95 < 800 ms).

Caution

k6’s http.post buffers the full response body before returning, even for streaming endpoints. The TTFT recorded this way is the time from request to the first token appearing in the buffered body — it still correctly captures inference server latency but does not model the true client-streaming experience where the UI renders the first token immediately. For true byte-by-byte streaming measurement, use k6’s experimental http.asyncRequest with a streaming body reader (k6 v0.47+).

Upload and run in MaxoPerf

1. Save the script as llm-streaming.js and upload it to your test’s Files tab as the Entrypoint with the k6 engine badge.

2. Add the LLM_API_KEY secret under Settings → Secrets.

3. In the Configuration tab set Duration to 8m and Virtual users to 15.

4. Add failure criteria: llm_ttft_ms p95 > 800 ms → fail and Error rate > 2% → fail.

5. Click Run. On the Overview tab, the custom metric llm_ttft_ms appears alongside the standard http_req_duration chart.

How to read streaming results

Metric	What to look for
http_req_duration p95	Full streaming latency — should be roughly TTFT + (output_tokens × TPOT). Rising without plateau means the server’s generation queue is growing.
llm_ttft_ms p95	TTFT proxy. Under 500 ms is excellent; 500–1000 ms is acceptable for most chat UIs; above 1 s degrades perceived responsiveness noticeably.
llm_output_tokens total	Cross-check: output_tokens / (test_duration_s × VUs) gives tokens/sec per VU — a proxy for GPU throughput.
Error rate	429s indicate quota; 504s indicate inference server overload.

Do / don’t

Do	Don’t
Fix max_tokens to control generation length and keep latency comparable	Let max_tokens vary — latency becomes a function of output length, not load
Emit TTFT as a custom Trend metric so it appears in results	Rely only on http_req_duration for streaming — it conflates TTFT and generation time
Set a generous HTTP timeout (60 s) — streaming 512 tokens at 30 tokens/s takes ~17 s	Use the k6 default 10 s timeout — it fires before long generations complete
Record which VU concurrency level saturates TTFT beyond your budget	Treat TTFT and end-to-end latency as interchangeable for streaming endpoints

Where to go next

• LLM performance and load testing — the baseline non-streaming load test.
• Inference cost and token budget testing — use the llm_output_tokens counter to project token cost under this load.
• AI API scalability: stress, spike, and soak — how TTFT degrades under stress and during soak runs.
• k6 scripts on Maxoperf — k6 engine reference on the Maxoperf platform.