RAG pipeline load testing

A retrieval-augmented generation (RAG) endpoint chains at least two steps that each add latency: a vector-database retrieval (nearest-neighbour search over embeddings) and an LLM generation call (with the retrieved context as the prompt). Under concurrency, either one can saturate on its own. A vector DB that handles 1 000 RPS in isolation can become the bottleneck at 50 concurrent RAG requests if it is not sized for parallel index scans. This page shows how to load-test the full RAG pipeline as one HTTP endpoint in MaxoPerf.

Before you start

Read LLM performance and load testing. RAG load testing uses the same HTTP POST mechanics.
Read Cookbook: CSV data-driven test. A realistic RAG load test uses a CSV file of real user queries, not one repeated prompt.
Your RAG endpoint should accept a user query string and return a generated answer (or at least a 200 status with a response body). MaxoPerf cannot see the retrieval and generation steps inside it. You measure the end-to-end latency of the full pipeline.

Where latency accrues in a RAG pipeline

User request
    │
    ▼
[ Query embedding ]      ← 10–100 ms  (embedding model inference)
    │
    ▼
[ Vector DB search ]     ← 5–200 ms   (ANN search; grows with index size and concurrency)
    │
    ▼
[ Context assembly ]     ← < 10 ms    (prompt construction; usually negligible)
    │
    ▼
[ LLM generation ]       ← 500 ms–10 s (dominant latency for long outputs)
    │
    ▼
Response

Under light load, vector DB search is fast (< 50 ms) and LLM generation takes most of the time. Under concurrent load, vector DB searches can queue, especially with CPU-bound approximate-nearest-neighbour (ANN) indexes. Once you have confirmed LLM generation capacity, watch whether the vector DB becomes the secondary bottleneck.

Realistic query CSV

If every iteration sends the same prompt, some vector databases answer from cache (Weaviate, Pinecone and Qdrant all cache recent queries), and the results look better than they are. Use a CSV of real or realistic queries:

query
"What are the return policies for international orders?"
"How do I reset my API key?"
"Can I export data to CSV from the dashboard?"
"What SLA does the enterprise plan offer?"
"How are embeddings indexed for multilingual content?"

Upload this as a test asset named queries.csv alongside your entrypoint file.

Taurus YAML with CSV query parameterisation

execution:
  - concurrency: 30          # 30 simultaneous RAG requests
    ramp-up: 3m              # ramp slowly — vector DB needs warm-up
    hold-for: 10m            # 10-minute plateau for vector-DB saturation signals
    scenario: rag-load

scenarios:
  rag-load:
    data-sources:
      - path: queries.csv     # uploaded as test asset
        variable-names: query
        random-order: true    # shuffle queries to prevent cache bias
    requests:
      - label: rag-query
        url: https://api.example.com/v1/rag/query
        method: POST
        headers:
          Content-Type: application/json
          Authorization: "Bearer ${RAG_API_KEY}"
        body: >
          {
            "query": "${query}",
            "top_k": 5,
            "max_tokens": 512,
            "stream": false
          }
        assert:
          - contains:
              subject: http-code
              value: '200'
          - contains:
              subject: body
              value: '"answer"'

Key points:

random-order: true shuffles the CSV row order per VU iteration. Without it, all VUs would send the same query at the same moment and inflate the vector-DB cache hit rate.
top_k: 5 sets how many document chunks are retrieved. Higher values give better answers but raise the vector-DB scan cost and the context-window token count (→ higher generation latency and cost).
concurrency: 30. RAG endpoints are more latency-sensitive than pure LLM endpoints because they chain two subsystems. Start at 20–30 VUs.

k6 version with CSV data and custom metrics

import http from 'k6/http';
import { check } from 'k6';
import { SharedArray } from 'k6/data';
import { Trend } from 'k6/metrics';
import papaparse from 'https://jslib.k6.io/papaparse/5.1.1/index.js';

const queries = new SharedArray('queries', function () {
  return papaparse.parse(open('./queries.csv'), { header: true }).data;
});

const ragLatency = new Trend('rag_e2e_ms', true);

export const options = {
  stages: [
    { duration: '3m', target: 30 },
    { duration: '10m', target: 30 },
    { duration: '2m', target: 0 },
  ],
  thresholds: {
    http_req_duration: ['p(95)<8000'],   // 8 s end-to-end for a RAG response
    http_req_failed:   ['rate<0.03'],
    rag_e2e_ms:        ['p(95)<8000'],
  },
};

const ENDPOINT = 'https://api.example.com/v1/rag/query';
const API_KEY  = __ENV.RAG_API_KEY;

export default function () {
  // Pick a random query from the CSV on each iteration
  const q = queries[Math.floor(Math.random() * queries.length)];
  const startMs = Date.now();

  const res = http.post(ENDPOINT, JSON.stringify({
    query:      q.query,
    top_k:      5,
    max_tokens: 512,
    stream:     false,
  }), {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${API_KEY}`,
    },
    timeout: '45s',
  });

  ragLatency.add(Date.now() - startMs);

  check(res, {
    'status 200':    (r) => r.status === 200,
    'has answer':    (r) => r.json('answer') !== undefined,
    'not throttled': (r) => r.status !== 429,
  });
}

How to create and run in MaxoPerf

Save the Taurus YAML as rag-load.yml (or the k6 script as rag-load.js) and prepare queries.csv with at least 200 representative queries.
In the MaxoPerf console, open Tests → New test. Name it rag-pipeline-30vu.
Open the Files tab. Upload rag-load.yml (or rag-load.js) as the Entrypoint and queries.csv as a Test asset.
Open Settings → Secrets and add RAG_API_KEY.
Open the Configuration tab. Set Virtual users to 30, Ramp-up to 3m, and Duration to 15m.
Add failure criteria: p95 latency > 8000 ms → fail and Error rate > 3% → fail.
Click Run and watch the latency chart. A RAG endpoint usually shows two phases: an early plateau (vector DB fast, LLM taking most of the time), then a latency climb once the vector DB saturates.

How to read the results

Signal	What it means
Latency plateau in the first 2–3 minutes, then climbs	Vector DB is warming up (building ANN graph cache). Normal; read the plateau, not the ramp.
p95 latency climbs linearly with VUs	The vector DB or LLM is saturating. The VU count where the climb starts tells you which.
High p99 / p95 divergence	Outlier queries with large retrieved contexts (many tokens → slow generation). Check `top_k` and chunk sizes.
429 errors	LLM API rate limit hit. Reduce concurrency or increase quota.
503 / 504 errors	RAG backend is overloaded, usually in the LLM layer. Check vector-DB health metrics too.

Cache effects and how to avoid them

Most production vector databases cache recent query embeddings and search results. In a load test with repeated queries, the cache hit rate is higher than real traffic would produce, so the result looks better than it is. To avoid this:

Use at least 200 distinct queries in your CSV (or 500+ for a large-scale test).
Set random-order: true (Taurus) or use Math.random() selection (k6) so queries reach VUs in random order.
If the system has explicit caching layers (Redis, CDN), run one test with the cache warm and one with the cache cold (flush the cache before the run). The two runs give you the best and worst latency.

Where to go next

Inference cost and token budget testing: injected RAG context raises input-token cost a lot.
AI API scalability: stress, spike, and soak: stress the RAG endpoint to find the saturation knee.
AI load test failure criteria: gate the RAG pipeline run with p95 and error-rate thresholds.
Cookbook: CSV data-driven test: how to upload and use CSV test data in MaxoPerf.