RAG pipeline load testing

A retrieval-augmented generation (RAG) endpoint combines at least two latency-bearing steps: a vector-database retrieval (nearest-neighbour search over embeddings) and an LLM generation call (using the retrieved context as the prompt). Under concurrency, these two subsystems can saturate independently — a vector DB that handles 1 000 RPS in isolation may become the bottleneck at 50 concurrent RAG requests if it is not provisioned for the index-scan parallelism. This page shows how to load-test the full RAG pipeline as one HTTP endpoint in MaxoPerf.

Before you start

Read LLM performance and load testing — RAG load testing uses the same HTTP POST mechanics.
Read Cookbook: CSV data-driven test — a realistic RAG load test uses a CSV file of real user queries, not a single repeated prompt.
Your RAG endpoint should accept a user query string and return a generated answer (or at minimum a 200 status with a response body). Internal retrieval and generation are opaque to MaxoPerf — you are measuring the end-to-end latency of the full pipeline.

Where latency accrues in a RAG pipeline

User request
    │
    ▼
[ Query embedding ]      ← 10–100 ms  (embedding model inference)
    │
    ▼
[ Vector DB search ]     ← 5–200 ms   (ANN search; grows with index size and concurrency)
    │
    ▼
[ Context assembly ]     ← < 10 ms    (prompt construction; usually negligible)
    │
    ▼
[ LLM generation ]       ← 500 ms–10 s (dominant latency for long outputs)
    │
    ▼
Response

Under light load, vector DB search is fast (< 50 ms) and LLM generation dominates. Under concurrent load, vector DB search can queue — especially for approximate-nearest-neighbour (ANN) indexes that are CPU-bound. Watch for the vector DB component becoming the secondary bottleneck after you have confirmed LLM generation capacity.

Realistic query CSV

Using the same prompt every iteration produces artificially cached results in some vector databases (Weaviate, Pinecone, Qdrant all cache recent queries). Use a CSV of real or representative queries:

query
"What are the return policies for international orders?"
"How do I reset my API key?"
"Can I export data to CSV from the dashboard?"
"What SLA does the enterprise plan offer?"
"How are embeddings indexed for multilingual content?"

Upload this as a test asset named queries.csv alongside your entrypoint file.

Taurus YAML with CSV query parameterisation

execution:
  - concurrency: 30          # 30 simultaneous RAG requests
    ramp-up: 3m              # ramp slowly — vector DB needs warm-up
    hold-for: 10m            # 10-minute plateau for vector-DB saturation signals
    scenario: rag-load

scenarios:
  rag-load:
    data-sources:
      - path: queries.csv     # uploaded as test asset
        variable-names: query
        random-order: true    # shuffle queries to prevent cache bias
    requests:
      - label: rag-query
        url: https://api.example.com/v1/rag/query
        method: POST
        headers:
          Content-Type: application/json
          Authorization: "Bearer ${RAG_API_KEY}"
        body: >
          {
            "query": "${query}",
            "top_k": 5,
            "max_tokens": 512,
            "stream": false
          }
        assert:
          - contains:
              subject: http-code
              value: '200'
          - contains:
              subject: body
              value: '"answer"'

Key points:

random-order: true — shuffles the CSV row order per VU iteration to avoid serving the same neighbour query to all VUs simultaneously, which would inflate vector-DB cache hit rate.
top_k: 5 — controls how many document chunks are retrieved. Higher values produce better answers but increase vector-DB scan cost and context-window token count (→ higher generation latency and cost).
concurrency: 30 — RAG endpoints are more latency-sensitive than pure LLM endpoints because they chain two subsystems. Start at 20–30 VUs.

k6 version with CSV data and custom metrics

import http from 'k6/http';
import { check } from 'k6';
import { SharedArray } from 'k6/data';
import { Trend } from 'k6/metrics';
import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';

const queries = new SharedArray('queries', function () {
  return papaparse.parse(open('./queries.csv'), { header: true }).data;
});

const ragLatency = new Trend('rag_e2e_ms', true);

export const options = {
  stages: [
    { duration: '3m', target: 30 },
    { duration: '10m', target: 30 },
    { duration: '2m', target: 0 },
  ],
  thresholds: {
    http_req_duration: ['p(95)<8000'],   // 8 s end-to-end for a RAG response
    http_req_failed:   ['rate<0.03'],
    rag_e2e_ms:        ['p(95)<8000'],
  },
};

const ENDPOINT = 'https://api.example.com/v1/rag/query';
const API_KEY  = __ENV.RAG_API_KEY;

export default function () {
  // Pick a random query from the CSV on each iteration
  const q = queries[Math.floor(Math.random() * queries.length)];
  const startMs = Date.now();

  const res = http.post(ENDPOINT, JSON.stringify({
    query:      q.query,
    top_k:      5,
    max_tokens: 512,
    stream:     false,
  }), {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${API_KEY}`,
    },
    timeout: '45s',
  });

  ragLatency.add(Date.now() - startMs);

  check(res, {
    'status 200':    (r) => r.status === 200,
    'has answer':    (r) => r.json('answer') !== undefined,
    'not throttled': (r) => r.status !== 429,
  });
}

How to create and run in MaxoPerf

Save the Taurus YAML as rag-load.yml (or the k6 script as rag-load.js) and prepare queries.csv with at least 200 representative queries.
In the MaxoPerf console, open Tests → New test. Name it rag-pipeline-30vu.
Open the Files tab. Upload rag-load.yml (or rag-load.js) as the Entrypoint and queries.csv as a Test asset.
Open Settings → Secrets and add RAG_API_KEY.
Open the Configuration tab. Set Virtual users to 30, Ramp-up to 3m, and Duration to 15m.
Add failure criteria: p95 latency > 8000 ms → fail and Error rate > 3% → fail.
Click Run and watch the latency chart. A RAG endpoint shows a characteristic two-phase ramp: early-plateau (vector DB fast, LLM dominating) transitioning to a latency climb when the vector DB saturates.

How to read the results

Signal	What it means
Latency plateau in the first 2–3 minutes, then climbs	Vector DB is warming up (building ANN graph cache). Normal; evaluate the plateau, not the ramp.
p95 latency climbs linearly with VUs	The vector DB or LLM is saturating. Identify which by observing at what VU count the climb begins.
High p99 / p95 divergence	Outlier queries with large retrieved contexts (many tokens → slow generation). Check `top_k` and chunk sizes.
429 errors	LLM API rate limit hit. Reduce concurrency or increase quota.
503 / 504 errors	RAG backend is overloaded — typically the LLM layer, but check vector-DB health metrics too.

Cache effects and how to avoid them

Most production vector databases cache recent query embeddings and search results. Under a load test with repeated queries, cache hit rate will be artificially high, giving you a falsely optimistic result. Mitigate this:

Use at least 200 distinct queries in your CSV (or 500+ for a large-scale test).
Set random-order: true (Taurus) or Math.random() selection (k6) to distribute queries across VUs unpredictably.
If you are testing a system with explicit caching layers (Redis, CDN), run one test with the cache warm and one with the cache cold (cache-flush before run) to bound the latency range.

Where to go next

Inference cost and token budget testing — RAG context injection inflates input-token cost significantly.
AI API scalability: stress, spike, and soak — stress the RAG endpoint to find the saturation knee.
AI load test failure criteria — gate the RAG pipeline run with p95 and error-rate thresholds.
Cookbook: CSV data-driven test — full guide to uploading and using CSV test data in MaxoPerf.