Skip to content

Choosing safe test environments

import { Aside } from ‘@astrojs/starlight/components’;

Load testing puts a system under stress on purpose. In the wrong environment, or without proper authorization, that stress can cause real outages, corrupt production data, send unwanted emails to real users, or trigger rate limits on third-party APIs you did not mean to hit. Choosing the environment and getting authorization are the first decisions in every load test design.

The choice between staging and production depends on what you are trying to learn and what you can afford to risk.

FactorStagingProduction
Risk to real usersNone (no real traffic)High: load test traffic competes with real users
Data integritySafe: test data can be resetRisky: test data may pollute real records
External APIsUsually mocked or sandboxedReal: may trigger costs, emails, notifications
RealismLower: caches colder, data smallerHigher: real traffic patterns, full data volume
Authorization neededSimpler: usually just your teamRequires cross-team coordination and change control

Staging is the default environment for performance testing, and MaxoPerf is built for it. Use staging when:

  • You are developing and iterating on test scripts.
  • You are running regression tests in CI.
  • You have not yet established a production performance baseline.
  • Any risk of service degradation is unacceptable.

The trade-off: staging results may not predict production behavior exactly. The gap grows when staging databases are much smaller (fewer cache misses, different query plans) or when staging hardware is heavily under-provisioned.

Sometimes production load testing is the only way to get accurate results. This is common for systems where staging differs too much from production to predict anything. Before you test in production:

  1. Notify all stakeholders, including on-call teams and the business (marketing, support).
  2. Schedule during a low-traffic window, such as nights or weekends.
  3. Start with a fraction of the target load. Ramp slowly and monitor.
  4. Have an abort plan. Know exactly how you will stop the MaxoPerf run if real users are affected.
  5. Isolate traffic if possible. Use a separate subdomain, a canary target, or a shadow traffic pattern so test traffic does not mix with real user traffic in your logs and analytics.

Managed cloud locations vs private locations

Section titled “Managed cloud locations vs private locations”

MaxoPerf offers two types of runner locations:

  • Managed cloud locations: MaxoPerf’s own runner infrastructure in AWS/GCP regions. Traffic originates from known MaxoPerf IP ranges. Use for external-facing services. See the Private locations concept page for the full list of managed regions.
  • Private locations (Outpost): your own runners, deployed in your infrastructure, behind your firewall. Traffic originates from inside your network. Use for internal services, behind-firewall targets, and VPN-only endpoints. See Private datacenter Outpost test for the setup recipe.

For internal APIs (accessible only from inside the corporate network or VPC), managed cloud locations cannot reach the target. You need private locations.

Incidents often come from a less obvious place. Your application under load may call third-party APIs, send emails, charge payment methods, or trigger webhooks, and a load test multiplies all of these.

Before you run a load test in any environment, audit your scenario for:

  • External API calls: payment processors, email providers, SMS gateways, analytics SDKs. These may have rate limits, per-request costs, or real-world side effects. Replace them with mocks or sandbox accounts in staging.
  • Email and notifications: a 1 000-VU test where each VU triggers a welcome email tries to send 1 000 emails. Make sure staging email senders are sandboxed.
  • Database writes: writes to a shared database can corrupt test data or affect other teams’ tests. Use isolated schemas or test-data namespaces.
  • Webhooks to external systems: if your system sends webhooks to CRM, ticketing, or ERP systems, your load test will trigger them in bulk.

Every MaxoPerf load test should be authorized at two levels:

  1. Technical authorization: you have confirmed that the target host (IP, domain, or internal endpoint) is one you are permitted to test. If the target is behind a WAF or a DDoS-protection service, allowlist MaxoPerf’s IP ranges for the test window so you avoid false positives.

  2. Organizational authorization: you have notified the target team, your on-call rotation, and any stakeholders who would be paged or affected. A brief Slack message and a calendar invite is usually enough for a staging test. Production usually calls for a formal change ticket.

DoDon’t
Use staging as the default environment for all development-phase and CI performance tests.Assume staging results predict production exactly. Document the known differences.
Get explicit authorization from the system owner before running any load test.Run load tests at “low enough” VU counts without authorization, assuming small tests are harmless.
Audit the scenario for third-party calls, email sends, and payment events before running at scale.Run a load test that exercises payment flows without confirming sandbox mode is active.
Use private locations (Outpost) for internal or behind-firewall targets.Try to reach internal services from managed cloud locations. They are not reachable from there.
Have an abort plan: know how to stop the MaxoPerf run quickly if something goes wrong.Start a 2-hour soak test in production without a monitoring dashboard and an on-call engineer watching.