A year of load failures: what broke, why, and how to test for it

Between June 2025 and June 2026, load broke things — repeatedly, expensively, and in patterns that any team building software for the internet should recognise. Game launches, Black Friday, a live premiere, a government results portal, a payday window: in each case, a real system that had passed normal testing collapsed under traffic that was, in hindsight, entirely predictable.

This post is a year-in-review. It collects twenty incidents from that period, organises them under ten failure patterns, and maps each pattern to the kind of load test that would have surfaced the risk before users found it. The incidents involve companies of every size — from government agencies to large cloud providers — and the lessons are not unique to any of them.

If you are new to performance testing, treat this as a field guide to what can go wrong and why. If you already run load tests, use it as a checklist: is your test portfolio covering all ten patterns?

The ten failure classes

Every outage in this series falls into one of these ten categories. Each links to the academy test type that best addresses it.

Flash-sale concurrency. A small inventory window triggers a rush of simultaneous buyers that overwhelms the checkout and inventory-decrement path. No queue or rate-limiting protects the write path. → Spike test

Scheduled-release spike. Everyone knows the exact release minute. Without pre-orders to spread demand, all buyers arrive at once, concentrating the entire purchase or authentication flow into seconds. → Spike test

Thundering herd. A large population of users all attempt to log in or connect simultaneously — at a launch or immediately after a brief outage ends. The auth layer, connection pools, or session store saturates before any warm-up can occur. → Spike test · Breakpoint/capacity test

Live-event peak. A broadcast or sporting event delivers a hard concurrency cliff at kickoff with no time-shift — every viewer hits play at the same second. The system needs to sustain that peak, not just survive the ramp. → Spike test · Soak/endurance test

Soak-growth collapse. The system handles the launch spike well but gradually degrades as concurrency keeps climbing over hours or days — a symptom of slow resource leaks, pool exhaustion, or cache pressure that only a short test misses. → Soak/endurance test

Retry storm. A brief disruption causes clients to reconnect simultaneously. If the recovery path lacks backoff and jitter, the reconnection wave is itself a denial-of-service attack, overloading the very systems trying to recover. → Stress test

Capacity ceiling. Traffic grows past the system’s real maximum — because autoscaling was not configured, replicas fell behind, or load shedding was absent. The ceiling was never measured, so it was never raised. → Breakpoint/capacity test · Scalability test

Dependency contention. A slow or temporarily degraded dependency — a database, a storage layer, an authentication backend — causes lock contention or connection exhaustion to cascade into the wider system. → Stress test

Config-limit cliff. A hard-coded size, feature-count, or quota limit is reached by an oversized input. The system does not degrade gracefully — it panics or crashes entirely. → Configuration test · Volume test

Seasonal demand spike. A known calendar event — tax season, exam results, payday — drives predictable peak traffic that the system was not sized for, often because it was never tested at that volume before the event. → Spike test · Volume test

The incident index

Flash-sale concurrency

C01 — The Odyssey IMAX onsale crash — Extreme demand for a limited run of 70mm screens sent AMC’s site into hour-long virtual queues and Fandango into error messages; AMC’s CEO described it as the highest first-day studio-film sales since 2022 — all concentrated into a single buy window.

Scheduled-release spike

C05 — Hollow Knight: Silksong storefront crash — No pre-orders meant 535,000 Steam players all hit the purchase and authentication flow at the launch minute, crashing Steam, the PlayStation Store, eShop, and Xbox at once.

C07 — Stranger Things S5 premiere outage — A synchronised global play-start spike on Netflix outpaced resource allocation for roughly twenty minutes; the co-creator noted that a 30% bandwidth increase still was not enough.

Thundering herd

C03 — Shopify Cyber Monday auth outage — On the highest-concurrency merchant day of the year, Shopify’s login, admin, POS, and API login flows went down for around six hours — a shared authentication chokepoint under simultaneous merchant load.

C04 — Battlefield 6 launch queues — Queues of up to 500,000 players formed at launch when simultaneous login demand outpaced provisioned capacity; EA scaled live and added admission queues to protect the experience.

C18 — SSA “My Social Security” portal crashes — A new anti-fraud check introduced earlier in the auth flow meant many more users hit authentication simultaneously; by public reporting, the system was not tested at high user volume before launch — the textbook missing load test.

Live-event peak

C08 — Netflix NFL Christmas streaming issues — Buffering, quality drops, and casting failures under massive concurrent live-sports load; a recurring challenge for any system that must sustain peak simultaneously across millions of viewers with no time-shift.

Soak-growth collapse

C06 — ARC Raiders server overload — The game passed a pre-launch server stress test and ran clean for several days, then login queues, matchmaking, and voice broke as concurrency kept climbing — the kind of gradual degradation only a long soak test reveals.

Retry storm

C09 — AWS US-EAST-1 cascade — A DNS issue triggered a mass simultaneous reconnection wave that overloaded the EC2 control plane in what the AWS postmortem called “congestive collapse” — the recovery itself became the load event.

C11 — Google Cloud us-central1 retry flood — A crash-looping service was stabilised globally in about forty minutes, but the us-central1 region’s recovery took three hours because clients retried without randomised exponential backoff, continuously reloading the datastore.

Capacity ceiling

C12 — GitHub sustained capacity failures — A multi-month cluster of outages attributed largely to traffic growing faster than capacity; database read replicas fell behind under peak read load, and some services could not shed load from high-volume clients.

C14 — Slack routing-config scale limit — Infrastructure growth silently outpaced static routing configurations, so routing updates stopped reaching the web layer and clients lost live routing data — a ceiling that was never measured until it was hit.

C15 — OpenAI/ChatGPT extended outage — Over ten hours and across many service components; no detailed public root-cause analysis was published, but AI inference capacity under heavy demand was cited as a contributing factor — an illustrative story about needing headroom and graceful 429 shedding.

Dependency contention

C13 — Clerk Cloud SQL migration contention — An unannounced live database migration spiked storage latency, increased lock contention, saturated compute, and caused a wave of 429s until the migration finished — the second such failure in seven months.

C19 — Fiserv/Zelle banking outage — A change at a shared core-banking platform cascaded across hundreds of institutions and more than sixty apps including Zelle, locking accounts and failing transfers during a Friday payday window.

Config-limit cliff

C10 — Cloudflare Bot Management crash — A database configuration change caused a data file to grow to more than twice its expected size, exceeding a hard feature-count limit; the proxy panicked on the oversized input instead of degrading gracefully, taking down Cloudflare’s core proxy globally for around six hours.

Seasonal demand spike

C02 — Best Buy Black Friday crash — The site and app buckled under the morning surge, with Downdetector reporting over 1,800 complaints; no official root-cause was published, but the timing matched the peak shopping window exactly.

C16 — NEET UG 2025 results-day overload — Over two million candidates attempted to download scorecards simultaneously on results day; the portal returned 503s, blank pages, and failed logins — a fully predictable spike that could have been sized and tested weeks in advance.

C17 — IRS “Where’s My Refund?” outage — The site went down mid filing-surge in what was officially described as a maintenance window; the lesson is that even planned downtime must account for seasonal concurrency.

C20 — UK payday banking outages — Multiple UK bank apps failed on a payday Friday; experts pointed to shared third-party dependencies under month-end peak transaction volume — a predictable calendar spike with fragile dependency handling.

How to start testing for these patterns

The incidents above cluster around a handful of testing gaps. Here is how to close each one.

If you have never load-tested before: start with a load test at your expected peak concurrency. Hold it for long enough to see steady-state behaviour — at least fifteen to thirty minutes. If nothing breaks, you have a baseline. If something breaks before you reach peak, you have your first finding.

For spike and flash-sale scenarios: run a spike test — ramp from baseline to several times peak almost instantly, hold briefly, then drop. Watch where error rate departs from zero and where latency crosses your threshold. Target the specific path under pressure: checkout, authentication, or inventory write, not just the homepage.

For thundering-herd and auth-under-load: spike the login and session path specifically, not just read-path traffic. Pair it with a breakpoint test to find the ceiling. If you have a new auth flow shipping, test it at the volume of the first deployment before the deployment.

For slow-burn and soak-growth scenarios: run a soak/endurance test at and beyond projected peak for hours, not minutes. A test that passes in five minutes may fail at the two-hour mark as connection pools exhaust or memory pressure accumulates.

For retry-storm and recovery scenarios: stress test the reconnection path — model many clients reconnecting simultaneously and observe whether throughput collapses or degrades gracefully. Check that your backoff has jitter so reconnection waves do not arrive in lockstep.

For capacity ceiling and scaling gaps: run a scalability test — step load up in increments until you find the knee of the curve. Record the max sustainable throughput and which subsystem fails first. Schedule recurring runs so you know when growing traffic approaches the old ceiling.

For dependency-contention scenarios: stress test while the dependency is degraded. In a staging environment, inject elevated database or storage latency and observe when lock contention and error rates increase.

For config-limit and boundary scenarios: run configuration and volume tests with oversized inputs — payload sizes, feature counts, or record volumes at and above the documented limit. Confirm that the system returns a clean error rather than crashing.

For seasonal and calendar peaks: size your spike to the known concurrency of the event — population multiplied by your concurrency assumption, with headroom. Run it several weeks before the date so you have time to act on the results. Schedule a rerun closer to the event to catch any changes since the first test.

MaxoPerf lets you run all of these patterns using Taurus, JMeter, or k6 scenarios from managed cloud locations or private runners, review results and run artifacts in a shared workspace, and schedule recurring runs so load testing becomes a continuous practice rather than a pre-launch scramble. A good starting point is the use-cases/ci-cd-performance-gates/ page if you want to attach a load gate to your release pipeline.

Key takeaways

• Traffic peaks are predictable more often than not. Black Friday, a game launch, an exam results page, a live premiere — each of these was on a calendar. The systems that failed were not tested at the traffic level the calendar implied.
• The test must target the right path. Many teams load-test the read path and miss the auth or write path. Several incidents in this series involved authentication chokepoints that storefront tests would never have found.
• Short tests miss slow degradation. A five-minute test can pass while a soak over hours reveals memory leaks, pool exhaustion, or queue back-pressure that only emerge under sustained load.
• Recovery is a load event. Two of the most severe outages here (the AWS and Google Cloud cases) were prolonged not by the initial failure but by the retry storm that followed it. If clients reconnect without backoff, recovery itself can trigger another outage.
• Hard limits need to be measured, not discovered. The Cloudflare incident is a clean example of a hard-coded ceiling that nobody tested against. Configuration and volume tests exist precisely to surface these before they surface in production.

Dive into any of the twenty incidents linked above for the full story — what happened, why it happened, and the specific test recipe that would have found the risk first.