Safe and ethical load testing

Load testing is one of the few development activities that can do real, immediate harm to a production system, or to systems you do not own. This page explains the authorization requirements, blast-radius controls and responsible practices that must be in place before you run any significant load test.

import { Aside } from ‘@astrojs/starlight/components’;

**Only run load tests against systems you are explicitly authorized to test.**

A load test against a system you are not authorized to test is a denial-of-service attack. That includes your own company’s production environment without SRE sign-off. It can cause real outages for real users, trigger incident responses, and in some jurisdictions may have legal consequences. Always get written authorization before you test any system you did not build and personally operate.

The authorization rule

Before you run any load test, answer yes to all three of these questions:

1. Do you own or have explicit written permission to load-test this system? “I think so” or “probably” is not a yes. Get explicit sign-off.

2. Have you notified everyone who will be affected by the test? That means your SRE/ops team, the teams that own dependent services, and your on-call rotation. In monitoring, a load test looks the same as a DDoS attack.

3. Do you have a runbook for stopping the test if it causes unintended impact? Know which button in MaxoPerf stops the test, and who to tell if you need to stop it at once.

If you cannot answer yes to all three, do not run the test.

Choosing a safe test environment

The safest place to load test is a dedicated performance environment: a staging or pre-production environment that mirrors production infrastructure but carries no real user traffic and no real data.

Environment	Safety level	Notes
Dedicated performance environment	Highest	Best choice; no real user impact
Shared staging	Medium	Coordinate with other teams; no prod data
Production (authorized, off-peak)	Low	Only with SRE sign-off + maintenance window
Production (uncoordinated)	Never	Do not do this

If you have no dedicated environment, see Choosing safe test environments in Foundations for the trade-offs between staging and production.

Blast radius controls

Even in a safe environment, a badly designed load test can spill over into systems you did not mean to stress. Apply these controls before every significant test:

Limit the scope of your test

• Test only the endpoints and flows you need to validate. A broad “test everything” approach widens the blast radius for no gain.

• Use MaxoPerf failure criteria with conservative thresholds to stop the test automatically if the error rate crosses a threshold early in the run:

  reporting:
    - module: passfail
      criteria:
        - subject: error-rate
          condition: '>'
          threshold: 10%   # stop if the error rate exceeds 10 %
          stop: true

Control the ramp rate

• A sudden jump to peak VUs can cause a cascade failure before you can react. Always ramp up gradually (minimum 2–3 minutes) so you can watch, and stop the test if the system degrades earlier than expected.

Protect dependent and third-party services

• Mock or stub third-party services that would receive real traffic during the test. Your load test should not send 10,000 real emails, charge 10,000 real payment transactions, or trigger 10,000 real SMS messages.
• Use test payment credentials (Stripe test mode, PayPal sandbox, etc.) when payment flows are in scope.
• Disable real notification delivery during the test window, or use a dedicated test account that routes notifications to a sink.

Know your MaxoPerf account limits and runner capacity

• Check your MaxoPerf plan limits for maximum concurrent VUs and maximum concurrent runners before a large test. If you exceed them, the test errors out instead of starting cleanly.
• If you are running a multi-region test, confirm that the locations you select are within your plan.

Data safety during testing

• Never use real production data (PII) as test data. See Test data dos and don’ts.
• Never create real financial transactions, real orders, or real user accounts during a load test. Use test credentials and test data that map to no-op handlers in your system.
• Purge test-generated records from your database after the test if your environment is shared with other teams.

Stopping a test safely

You can stop a running MaxoPerf test at any time from the run-detail page or from the account-level active runs view. If you see any of the following during a test, stop at once and investigate before you continue:

• Error rate climbing above your agreed blast-radius threshold.
• Alerting firing in your production monitoring system (if testing in or near production).
• A third-party service showing unexpected load in their dashboard.
• Your system becoming unresponsive to health checks.

The stop action in MaxoPerf sends a graceful termination to all runners. Runners drain in-flight requests (up to a configured drain timeout) before stopping.

Responsible disclosure

A load test can uncover a performance vulnerability: a crash at a specific load level, a memory leak that causes OOM kills, or an unauthenticated endpoint that is trivial to saturate. If you find one, follow your organization’s responsible disclosure process before you publish or share the findings publicly.

Checklist before testing a system you do not own

• Written authorization from the system owner is on file.
• SRE / operations team notified with test window and expected load.
• On-call rotation informed and available during the test window.
• Third-party services mocked, stubbed, or using test credentials.
• Failure criteria configured to auto-stop on early error-rate breach.
• Runbook for emergency stop is documented and distributed.
• Post-test cleanup plan for test-generated records is agreed.

Where to go next

• Choosing safe test environments: staging vs prod
• Pre-flight checklist: the full run checklist
• Common pitfalls and anti-patterns: anti-pattern #1 in detail
• Failure criteria pass/fail gates: auto-stop thresholds