Configuration test

A configuration test (sometimes called a comparison test) runs the same load profile against two or more different infrastructure or application configurations to measure the performance impact of a specific change. Examples: caching on vs off, connection pool size 10 vs 50, database index present vs absent, CDN enabled vs direct origin.

Before you start

• You can deploy the target system in two or more distinct configurations without changing the application code or test script.
• You have a load test that establishes a repeatable baseline.
• Both configurations use the same data set, network path, and infrastructure tier (same region, same machine size) so that the only variable is the configuration under test.

What is a configuration test?

A configuration test is a controlled experiment. Keeping everything else fixed, you change one variable at a time and observe the performance effect. This is the performance-testing equivalent of an A/B test.

Common configuration variables tested this way:

Variable	Configuration A	Configuration B
Query cache	Disabled	Enabled (Redis TTL 60 s)
Database connection pool	Size 10	Size 50
CDN / edge cache	Bypass	Enabled
JVM heap size	512 MB	2 GB
Database index	Missing	Present
HTTP keep-alive	Disabled	Enabled
Replica count	1	2

How to run a configuration test in MaxoPerf

Load profile

Parameter	Value
Virtual users (VUs)	Fixed (same as load test — e.g., 100 VUs)
Duration	15 min per run
Ramp-up	3 min (identical across runs)
Stop mode	Duration
Locations	Same location(s) across all runs

Console walk-through

1. Deploy Configuration A (for example, caching disabled).
2. Run your standard load test. Tag the run config=no-cache in the run’s Notes field.
3. Deploy Configuration B (caching enabled). Do not change the test definition.
4. Run the identical test again. Tag the run config=redis-cache.
5. Open either run and click Compare to select the other run as the baseline. The comparison view shows throughput, p95/p99 latency, and error rate deltas.

# Same YAML for both runs — only the target infrastructure changes
execution:
  - executor: jmeter
    concurrency: 100
    ramp-up: 3m
    hold-for: 12m
    scenario: api-config-test

scenarios:
  api-config-test:
    requests:
      - url: https://api.staging.example.com/v1/products
        label: list-products
      - url: https://api.staging.example.com/v1/users/profile
        label: get-profile

How to read the result

Use MaxoPerf’s run comparison view to quantify the effect of each configuration:

• Throughput delta — does Configuration B serve more requests per second? A 30 % throughput increase with caching enabled is a clear win.
• p95 latency delta — how much did p95 improve or worsen? A green chip (lower is better) confirms the configuration change helped.
• Error rate delta — ensure the configuration change did not introduce new errors.
• Consistency — run each configuration at least twice. If the two runs of the same configuration show significantly different results, the environment is not stable enough for comparison.

Do / don’t

Do	Don’t
Change one variable at a time	Change multiple configuration knobs between runs
Tag every run with the configuration it tested	Rely on memory to track which run tested which config
Run each configuration at least twice for consistency	Draw conclusions from a single run per configuration
Keep VU count, duration, and ramp-up identical across runs	Adjust the load profile between runs
Check that both environments are warmed up equally	Compare a cold-cache run against a warm-cache run

Where to go next

• Scalability test — compare performance across different replica counts.
• Baseline regression test — track performance changes across deploys, not just config changes.
• Cookbook: Comparing runs and baselines — a step-by-step guide to the run comparison feature.