Configuration test

A configuration test (sometimes called a comparison test) runs the same load profile against two or more infrastructure or application configurations. It measures what one specific change does to performance. Examples: caching on vs off, connection pool size 10 vs 50, database index present vs absent, CDN enabled vs direct origin.

Before you start

• You can deploy the target system in two or more distinct configurations without changing the application code or test script.
• You have a load test that establishes a repeatable baseline.
• Both configurations use the same data set, network path and infrastructure tier (same region, same machine size), so the configuration under test is the only variable.

What is a configuration test?

A configuration test is a controlled experiment. You keep everything else fixed, change one variable at a time, and watch what happens to performance. It is the performance testing version of an A/B test.

Variables teams often test this way:

Variable	Configuration A	Configuration B
Query cache	Disabled	Enabled (Redis TTL 60 s)
Database connection pool	Size 10	Size 50
CDN / edge cache	Bypass	Enabled
JVM heap size	512 MB	2 GB
Database index	Missing	Present
HTTP keep-alive	Disabled	Enabled
Replica count	1	2

How to run a configuration test in MaxoPerf

Load profile

Parameter	Value
Virtual users (VUs)	Fixed (same as load test, e.g., 100 VUs)
Duration	15 min per run
Ramp-up	3 min (identical across runs)
Stop mode	Duration
Locations	Same location(s) across all runs

Console walk-through

1. Deploy Configuration A (for example, caching disabled).
2. Run your standard load test. Tag the run config=no-cache in the run’s Notes field.
3. Deploy Configuration B (caching enabled). Do not change the test definition.
4. Run the identical test again. Tag the run config=redis-cache.
5. Open either run and click Compare to select the other run as the baseline. The comparison view shows throughput, p95/p99 latency, and error rate deltas.

# Same YAML for both runs — only the target infrastructure changes
execution:
  - executor: jmeter
    concurrency: 100
    ramp-up: 3m
    hold-for: 12m
    scenario: api-config-test

scenarios:
  api-config-test:
    requests:
      - url: https://api.staging.example.com/v1/products
        label: list-products
      - url: https://api.staging.example.com/v1/users/profile
        label: get-profile

How to read the result

Use MaxoPerf’s run comparison view to measure the effect of each configuration:

• Throughput delta. Check whether Configuration B serves more requests per second. A 30 % throughput increase with caching enabled is a clear win.
• p95 latency delta. Check how far p95 moved. A green chip (lower is better) confirms the change helped.
• Error rate delta. Make sure the change did not add new errors.
• Consistency. Run each configuration at least twice. If two runs of the same configuration give very different results, the environment is not stable enough to compare.

Do / don’t

Do	Don’t
Change one variable at a time	Change multiple configuration knobs between runs
Tag every run with the configuration it tested	Rely on memory to track which run tested which config
Run each configuration at least twice for consistency	Draw conclusions from a single run per configuration
Keep VU count, duration, and ramp-up identical across runs	Adjust the load profile between runs
Check that both environments are warmed up equally	Compare a cold-cache run against a warm-cache run

Where to go next

• Scalability test: compare performance across different replica counts.
• Baseline regression test: track performance changes across deploys as well as config changes.
• Cookbook: Comparing runs and baselines: a step-by-step guide to the run comparison feature.