Inference cost and token budget testing

Every LLM API call costs money in proportion to two numbers: input tokens (the prompt) and output tokens (the generated response). Under load, that cost multiplies across every concurrent user and every request. A load test that does not count tokens gives you latency data and no cost model. This page shows how to count tokens in your load tests and project monthly spend from the results.

Before you start

• Read LLM performance and load testing. Token cost analysis starts from the baseline load test.
• Read Streaming and token latency testing if you are using streaming responses (token counts come from SSE chunks).
• You need the per-token pricing for your model. Most providers publish this as a rate per 1,000 tokens (input and output separately).

Token accounting under load

A hosted LLM API charges per token in two buckets:

Bucket	Typical source	Notes
Input tokens	Prompt + system message	Fixed per request if you use the same prompt
Output tokens	Model-generated content	Varies if max_tokens is not fixed

The response body (for non-streaming calls) contains exact token counts in the usage field:

{
  "id": "chatcmpl-abc123",
  "choices": [...],
  "usage": {
    "prompt_tokens": 42,
    "completion_tokens": 218,
    "total_tokens": 260
  }
}

Extracting token counts in k6

Add a custom metric to your existing load-test script to record the tokens each request uses:

import http from 'k6/http';
import { check } from 'k6';
import { Trend, Counter, Rate } from 'k6/metrics';

const inputTokens  = new Counter('llm_input_tokens');
const outputTokens = new Counter('llm_output_tokens');
const costUsd      = new Trend('llm_cost_usd_per_req', true);

// Pricing — update to match your model and provider
const INPUT_PRICE_PER_1K  = 0.0005;  // USD per 1 000 input tokens  (e.g. gpt-4o-mini)
const OUTPUT_PRICE_PER_1K = 0.0015;  // USD per 1 000 output tokens

export const options = {
  stages: [
    { duration: '2m', target: 20 },
    { duration: '5m', target: 20 },
    { duration: '1m', target: 0 },
  ],
  thresholds: {
    http_req_duration: ['p(95)<4000'],
    http_req_failed:   ['rate<0.02'],
  },
};

const ENDPOINT = 'https://inference.example.com/v1/chat/completions';
const API_KEY  = __ENV.LLM_API_KEY;

const PAYLOAD = JSON.stringify({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Explain what load testing is in two paragraphs.' }],
  max_tokens: 256,
  temperature: 0.7,
  stream: false,
});

export default function () {
  const res = http.post(ENDPOINT, PAYLOAD, {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${API_KEY}`,
    },
    timeout: '30s',
  });

  check(res, { 'status 200': (r) => r.status === 200 });

  if (res.status === 200) {
    const body = res.json();
    const usage = body?.usage ?? {};

    const promptTok  = usage.prompt_tokens     ?? 0;
    const completeTok = usage.completion_tokens ?? 0;

    inputTokens.add(promptTok);
    outputTokens.add(completeTok);

    const reqCost = (promptTok  / 1000) * INPUT_PRICE_PER_1K
                  + (completeTok / 1000) * OUTPUT_PRICE_PER_1K;
    costUsd.add(reqCost);
  }
}

After a 5-minute plateau at 20 VUs, MaxoPerf will show:

• llm_input_tokens total: the sum of all prompt tokens used in the run.
• llm_output_tokens total: the sum of all output tokens generated.
• llm_cost_usd_per_req: p50/p95/max cost of a single request, in USD.

Worked cost example

Assume the 7-minute test (2 m ramp + 5 m plateau) at 20 VUs produces:

Metric	Observed value
Total requests completed	840
Avg input tokens/request	42
Avg output tokens/request	218
Model pricing (input)	$0.0005 / 1 k tokens
Model pricing (output)	$0.0015 / 1 k tokens

Cost per request:

(42 / 1000) × $0.0005  +  (218 / 1000) × $0.0015
= $0.0000210  +  $0.000327
= $0.000348 per request

Total test cost:

840 × $0.000348 = $0.29

Projecting monthly cost at this traffic level:

If 20 VUs at 5-minute steady state yields roughly 20 ÷ 2.5 s avg latency = 8 RPS, then monthly:

8 req/s × 86 400 s/day × 30 days = 20 736 000 requests/month
20 736 000 × $0.000348 = $7 216 / month

This monthly projection is the main result of a cost-under-load test. Compare it with your LLM budget before you launch a feature.

Prompt-size impact

Input token count is the variable you control most directly. Run your test at three prompt sizes to see how much cost moves with it:

Prompt variant	Approx input tokens	Cost per request	Monthly Δ
Short (1 sentence)	~20	$0.000337	baseline
Medium (2 paragraphs)	~150	$0.000402	+19%
Long (full document excerpt)	~1 000	$0.000677	+101%

A complex RAG system that injects 1 000-token context windows doubles the cost vs a simple chat prompt. See RAG pipeline load testing for the retrieval side.

Watching cost during a stress run

During an AI stress test, tokens used per unit of time climb with VUs until the model saturates. Watch for three phases:

• Below saturation: cost/second grows linearly with RPS, as expected.
• At saturation: RPS levels off but latency climbs. Cost/second levels off too (fewer completed requests), while latency p95 grows fast. You pay for queuing time, not useful output.
• After rate-limit (429) kicks in: failed requests are not billed, but retries are. If your application retries automatically on 429, the real cost can exceed your estimate.

Do / don’t

Do	Don’t
Fix max_tokens in every run; varying output tokens make cost comparisons meaningless	Let max_tokens be model-decided (null) in load tests
Extract and record the usage field from every response	Estimate token counts from string length. Tokenisation is model-specific and does not track character count
Project monthly cost from RPS × cost/request before launching	Wait until the first monthly invoice to discover cost at scale
Test several prompt sizes to measure how cost changes with prompt length	Assume the cost model is linear across all prompt lengths; context window overhead varies
Count retry cost when rate-limiting applies	Treat 429 responses as free; retries use quota and budget

Where to go next

• LLM performance and load testing: the baseline test this page builds on.
• Streaming and token latency testing: extract token counts from SSE chunks.
• RAG pipeline load testing: what injecting retrieved context does to cost.
• AI load test failure criteria: set a cost-per-request threshold as a failure criterion.
• AI API scalability: stress, spike, and soak: how cost behaves under saturation and soak.