Articles
Performance testing field notes
Practical guidance for teams moving from one-off load scripts to repeatable cloud and private-location performance workflows.
- A year of load failures: what broke, why, and how to test for it · load testing, outages, performance testing, spike testing, capacity planning
Twenty real outages, ten failure patterns, one year. A guided tour of what load does to software in production — and how to test before it happens to you.
- ARC Raiders passed its pre-launch stress test — then overloaded anyway. Here's the soak lesson. · load testing, soak-growth-collapse, gaming
ARC Raiders ran clean for days after launch, then login queues and matchmaking broke as concurrent players kept climbing. Reporting attributes this to sustained growth beyond a one-time peak. Embark did not confirm a root cause.
- AWS US-East-1 outage (October 2025): when recovery itself becomes the failure · load testing, retry-storm, cloud
A DNS race triggered the October 2025 AWS US-East-1 cascade, but it was the recovery overload — thousands of services reconnecting at once — that turned a blip into 15 hours of disruption.
- Battlefield 6 launch server queues — what 500,000 players waiting tells you about load testing · load testing, thundering-herd, gaming
On launch day EA queued over 500,000 concurrent players. Here's the thundering-herd pattern behind that decision and how to test for it before your next big release.
- Best Buy's Black Friday 2025 crash: what every retail team can learn about seasonal demand spikes · load testing, seasonal-demand-spike, e-commerce
Best Buy's site and app buckled under Black Friday morning traffic for roughly 90 minutes in November 2025. No official RCA was published — but the failure pattern is well understood.
- Clerk Cloud SQL migration outage — when a database operation becomes a load event · load testing, dependency-contention, saas
A routine storage migration turned into a 26-minute outage for Clerk in March 2026. The dependency-contention failure pattern it exposed applies to any team running a live database.
- Cloudflare outage (November 2025): the hard size limit that crashed a global proxy · load testing, config-limit-cliff, cloud
A database query change caused an oversized configuration file to exceed a hard internal limit, crashing Cloudflare's core proxy globally for ~3 hours in November 2025. The lesson is boundary testing before shipping config changes.
- Fiserv/Zelle banking outage 2025 — when a shared platform goes down, everyone goes down · load testing, dependency-contention, fintech
In 2025, a change at a Fiserv data center cascaded across a shared core-banking platform, taking Zelle and 60+ banking apps offline for over 12 hours during a payday window.
- GitHub capacity outages 2025–2026 — what a year of failures teaches about load testing · load testing, capacity-ceiling, saas
GitHub suffered dozens of capacity-related outages between 2025 and early 2026. Here's what caused them and how to test for the same failure pattern.
- Google Cloud outage (June 2025): the retry flood that extended a 40-minute fix into 3 hours · load testing, retry-storm, cloud
A null-pointer bug in new quota-policy code crash-looped Google Cloud's request-authorization service globally in June 2025. Most regions recovered in 40 minutes — but us-central1 spent 3 hours fighting a retry flood caused by missing backoff logic.
- Hollow Knight Silksong crashed Steam and three storefronts — the scheduled-release spike every publisher should test for · load testing, scheduled-release-spike, gaming
Silksong's no-pre-orders launch decision concentrated 535,000 day-one players into a single minute and brought down Steam, PS Store, and eShop. Here's how to test for it.
- IRS "Where's My Refund?" went down during peak filing season — what every team can learn · load testing, seasonal-demand-spike, government
In February 2026, the IRS refund tracker went offline mid-filing surge. The lesson is not the outage itself — it is that maintenance windows must be planned around load-tested seasonal peaks.
- NEET UG 2025 results day crash — what 2 million simultaneous logins look like · load testing, seasonal-demand-spike, education
On results day, 2M+ candidates hit neet.nta.nic.in at once, triggering 503/504 errors and blank scorecards. Here is what happened and how to test for it.
- Netflix NFL Christmas streaming issues — the live-event load problem with no time-shift · load testing, live-event-peak, streaming
Netflix viewers reported buffering and casting failures during NFL Christmas 2025. No confirmed RCA, but the live-event load pattern has a clear lesson.
- The Odyssey IMAX ticket onsale crash: what flash-sale concurrency really looks like · load testing, flash-sale-concurrency, e-commerce
When Nolan's The Odyssey went on sale, AMC and Fandango buckled under simultaneous demand for limited 70mm seats. Here's the failure pattern and how to test for it.
- OpenAI ChatGPT outage June 2025 — an illustrative AI-capacity story · load testing, capacity-ceiling, ai
A 10-plus-hour ChatGPT outage in June 2025 affected free, Plus, API, and Sora users. No official root-cause analysis was published. Here is what the pattern suggests — and how to test for it.
- Shopify's Cyber Monday 2025 outage: what a shared auth chokepoint looks like under peak merchant load · load testing, thundering-herd, e-commerce
On Cyber Monday 2025, Shopify's admin, POS, and API access went down for hours while merchant storefronts largely survived. The company officially attributed the issue to a login authentication flow problem.
- Slack outage May 2025 — when infrastructure growth outpaces static configuration · load testing, capacity-ceiling, saas
In May 2025, Slack's growth hit a silent limit in its routing configuration, cutting live connectivity for users for nearly two hours. Here's what the pattern means for any growing team.
- SSA "My Social Security" portal crashes — the cost of skipping a load test before launch · load testing, thundering-herd, government
In early 2025, the Social Security Administration portal crashed repeatedly after a software update. Reporting attributed it to staff not testing the software against high user volumes before launch.
- Stranger Things S5 premiere outage — when 30% extra capacity still isn't enough · load testing, scheduled-release-spike, streaming
Netflix pre-scaled ~30% for the Stranger Things S5 premiere but still went down for ~20 minutes. The spike-test lesson for every streaming team.
- UK banks payday outages 2025 — when month-end peak meets a fragile shared dependency · load testing, seasonal-demand-spike, fintech
In 2025, Lloyds, Halifax, TSB, Nationwide, and others failed on payday Fridays. The formal cause was not disclosed, but the pattern points to peak transaction volume colliding with a suspected shared dependency.
- API performance testing as a release gate · api performance, ci cd, release gates
A practical pattern for turning API load tests into CI/CD gates without hiding failures behind one average number.
- When private load generation beats public runners · byoc, private locations, load generation
Use private runner locations when the target is internal, regulated, latency-sensitive, or protected by network boundaries.
- Pair load tests with live observability · observability, run results, latency
What to watch during a load test: throughput, latency percentiles, errors, runner health, and the logs that explain failures.
- Why teams move load testing to the cloud · cloud load testing, managed runners, results
How managed runners, private locations, repeatable bundles, and shared run results make load testing easier to trust.