Inference cost and token budget testing

Every LLM API call has a cost that scales with two quantities: input tokens (the prompt) and output tokens (the generated response). Under load, this cost multiplies across every concurrent user and every request. Running a load test without tracking token consumption leaves you with latency data but no cost model — this page shows how to instrument token accounting in your load tests and project monthly spend from the results.

Before you start

• Read LLM performance and load testing — token cost analysis starts from the baseline load test.
• Read Streaming and token latency testing if you are using streaming responses (token counts come from SSE chunks).
• You need the per-token pricing for your model. Most providers publish this as a rate per 1,000 tokens (input and output separately).

Token accounting under load

A hosted LLM API charges per token in two buckets:

Bucket	Typical source	Notes
Input tokens	Prompt + system message	Fixed per request if you use the same prompt
Output tokens	Model-generated content	Varies if max_tokens is not fixed

The response body (for non-streaming calls) contains exact token counts in the usage field:

{
  "id": "chatcmpl-abc123",
  "choices": [...],
  "usage": {
    "prompt_tokens": 42,
    "completion_tokens": 218,
    "total_tokens": 260
  }
}

Extracting token counts in k6

Add a custom metric to your existing load-test script to capture token consumption per request:

import http from 'k6/http';
import { check } from 'k6';
import { Trend, Counter, Rate } from 'k6/metrics';

const inputTokens  = new Counter('llm_input_tokens');
const outputTokens = new Counter('llm_output_tokens');
const costUsd      = new Trend('llm_cost_usd_per_req', true);

// Pricing — update to match your model and provider
const INPUT_PRICE_PER_1K  = 0.0005;  // USD per 1 000 input tokens  (e.g. gpt-4o-mini)
const OUTPUT_PRICE_PER_1K = 0.0015;  // USD per 1 000 output tokens

export const options = {
  stages: [
    { duration: '2m', target: 20 },
    { duration: '5m', target: 20 },
    { duration: '1m', target: 0 },
  ],
  thresholds: {
    http_req_duration: ['p(95)<4000'],
    http_req_failed:   ['rate<0.02'],
  },
};

const ENDPOINT = 'https://inference.example.com/v1/chat/completions';
const API_KEY  = __ENV.LLM_API_KEY;

const PAYLOAD = JSON.stringify({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Explain what load testing is in two paragraphs.' }],
  max_tokens: 256,
  temperature: 0.7,
  stream: false,
});

export default function () {
  const res = http.post(ENDPOINT, PAYLOAD, {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${API_KEY}`,
    },
    timeout: '30s',
  });

  check(res, { 'status 200': (r) => r.status === 200 });

  if (res.status === 200) {
    const body = res.json();
    const usage = body?.usage ?? {};

    const promptTok  = usage.prompt_tokens     ?? 0;
    const completeTok = usage.completion_tokens ?? 0;

    inputTokens.add(promptTok);
    outputTokens.add(completeTok);

    const reqCost = (promptTok  / 1000) * INPUT_PRICE_PER_1K
                  + (completeTok / 1000) * OUTPUT_PRICE_PER_1K;
    costUsd.add(reqCost);
  }
}

After a 5-minute plateau at 20 VUs, MaxoPerf will show:

• llm_input_tokens total — sum of all prompt tokens consumed in the run.
• llm_output_tokens total — sum of all output tokens generated.
• llm_cost_usd_per_req — p50/p95/max cost per single request, in USD.

Worked cost example

Assume the 7-minute test (2 m ramp + 5 m plateau) at 20 VUs produces:

Metric	Observed value
Total requests completed	840
Avg input tokens/request	42
Avg output tokens/request	218
Model pricing (input)	$0.0005 / 1 k tokens
Model pricing (output)	$0.0015 / 1 k tokens

Cost per request:

(42 / 1000) × $0.0005  +  (218 / 1000) × $0.0015
= $0.0000210  +  $0.000327
= $0.000348 per request

Total test cost:

840 × $0.000348 = $0.29

Projecting monthly cost at this traffic level:

If 20 VUs at 5-minute steady state yields roughly 20 ÷ 2.5 s avg latency = 8 RPS, then monthly:

8 req/s × 86 400 s/day × 30 days = 20 736 000 requests/month
20 736 000 × $0.000348 = $7 216 / month

This monthly projection is the primary output of a cost-under-load test. Compare it against your LLM budget before launching a feature.

Prompt-size impact

Input token count is the variable you control most directly. Run your test at three prompt sizes to understand the sensitivity:

Prompt variant	Approx input tokens	Cost per request	Monthly Δ
Short (1 sentence)	~20	$0.000337	baseline
Medium (2 paragraphs)	~150	$0.000402	+19%
Long (full document excerpt)	~1 000	$0.000677	+101%

A complex RAG system that injects 1 000-token context windows doubles the cost vs a simple chat prompt. See RAG pipeline load testing for the retrieval side.

Watching cost during a stress run

During an AI stress test, token consumption per unit time climbs with VUs until the model saturates. The key observation is:

• Below saturation: cost/second scales linearly with RPS — expected.
• At saturation: RPS plateaus but latency climbs. Cost/second plateaus too (fewer completed requests), while the latency p95 blows out — you are paying for queuing time, not useful output.
• After rate-limit (429) kicks in: failed requests are not billed, but retries are. If your application auto-retries on 429, actual cost can exceed your model estimate.

Do / don’t

Do	Don’t
Fix max_tokens in every run — variable output tokens make cost comparisons meaningless	Let max_tokens be model-decided (null) in load tests
Extract and record the usage field from every response	Estimate token counts from string length — tokenisation is model-specific and not character-proportional
Project monthly cost from RPS × cost/request before launching	Wait until the first monthly invoice to discover cost at scale
Test multiple prompt sizes to quantify the sensitivity of your prompts	Assume the cost model is linear across all prompt lengths — context window overhead varies
Account for retry cost when rate-limiting is in play	Treat 429 responses as free — retries consume quota and budget

Where to go next

• LLM performance and load testing — the baseline test this page extends.
• Streaming and token latency testing — extract token counts from SSE chunks.
• RAG pipeline load testing — cost implications of context-window injection from retrieval.
• AI load test failure criteria — set a cost-per-request threshold as a failure criterion.
• AI API scalability: stress, spike, and soak — cost behaviour under saturation and soak.