Articles
Performance testing field notes
Practical guidance for teams moving from one-off load scripts to repeatable cloud and private-location performance workflows.
- Load Testing Authentication, Sessions, and Token Refresh Safely · authentication, sessions, load testing
Test login capacity separately from authenticated traffic and model expiry, refresh, session state, and credential safety without leaking secrets.
- How to Organize a k6 Test Suite for a Growing Engineering Team · k6, test maintenance, load testing
Organize a k6 suite with explicit scenarios, environment adapters, checks, shared primitives, and reviewable ownership.
- Think Time, Pacing, and Session Length: Making Virtual Users Behave Like People · virtual users, load modeling, user journeys
Separate think time from pacing and session length so virtual-user concurrency and throughput remain explainable in load tests.
- Performance Testing in CI/CD: PR, Nightly, and Release-Test Cadence · CI/CD, performance testing, release gates
Build a performance-testing cadence with a small PR guard, a broader nightly run, and representative release evidence.
- How to Read a Latency Histogram Without Fooling Yourself · latency, histograms, performance metrics
Read latency histograms by checking buckets, tails, multiple modes, aggregation, sample counts, and the workload behind every shape.
- Healthcare.gov’s Launch: What Incomplete End-to-End Testing Costs · incident analysis, end-to-end testing, capacity planning
The GAO's Healthcare.gov review shows how capacity, integration, pass criteria, requirements, and launch governance fail together.
- How to Debug a Load Test Before You Scale It · debugging, load testing, test data
Prove one user, one journey, assertions, data mutation, and cleanup before multiplying a load-test script into a confusing failure.
- Performance Thresholds That Catch Regressions Without Blocking Every Build · performance testing, CI/CD, release gates
Derive useful performance thresholds from user and service objectives, then add noise tolerance without hiding regressions.
- Arrival Rate vs Concurrency: Open and Closed Workload Models · load modeling, arrival rate, concurrency
Understand open and closed workload models, why slow systems change closed-loop demand, and how to avoid coordinated omission.
- The Pre-Run Load Testing Checklist That Prevents Misleading Results · load testing, test planning, reliability
Check authorization, data, telemetry, generator capacity, abort conditions, and recovery before every meaningful load test.
- Browser vs Protocol Load Testing: When You Need One—or Both · browser testing, protocol testing, load testing
Learn when protocol tests create scale, browser tests measure user experience, and a small browser cohort should accompany API load.
- How to Turn Production Analytics into Weighted User Journeys · analytics, load modeling, user journeys
Translate analytics into realistic load-test journeys while correcting for bots, retries, cache hits, seasonality, and missing client events.
- Building a Production-Like Test Environment Without Cloning Production · test environments, performance testing, capacity planning
Build a useful production-like performance environment by preserving bottleneck constraints, data shape, topology, and observability instead of cloning everything.
- The Ticketmaster Eras Tour On-Sale: What 3.5 Billion Requests Teach About Admission Control · incident analysis, admission control, spike testing
Ticketmaster's Eras Tour on-sale account separates human demand, bot amplification, queues, and admission control for safer load planning.
- Little’s Law for Load Testers: Concurrency, Throughput, and Time · load modeling, throughput, latency
Use Little’s Law to sanity-check concurrency, throughput, and response time in load-test plans without mistaking the estimate for a benchmark.
- Staging or Production? Where Should You Run Load Tests? · load testing, staging, production readiness
Choose staging or production for load testing with a risk-based ladder covering fidelity, safety, observability, and evidence limits.
- Why Average Response Time Lies Under Load · latency, performance metrics, load testing
Averages hide slow cohorts. Learn to read percentiles, distributions, throughput, and errors together when interpreting load-test results.
- k6 vs JMeter: Choosing for Your Team, Not a Benchmark · k6, JMeter, load testing, tool selection
Compare k6 and JMeter by maintenance, protocols, language, debugging, data, and execution needs instead of synthetic tool races.
- Performance Test Requirements: 12 Questions to Answer Before You Script · performance testing, test planning, quality engineering
Use 12 practical questions to define a performance test's decision, workload, data, observability, and pass criteria before scripting.
- Virtual Users vs Requests per Second: Which Load Model Should You Use? · load modeling, virtual users, throughput
Understand the difference between virtual users and requests per second, then choose the control variable that matches real demand.
- From Production Traffic to a Load-Test Plan: A Practical Workflow · load testing, performance testing, capacity planning
Turn production evidence into a load-test plan with explicit journeys, demand assumptions, safety limits, and measurable acceptance criteria.
- The Black Friday wall of shame: a decade of outages, and the tests that would have caught them · load testing, black friday, outages, e-commerce, spike testing, peak traffic
Best Buy, Costco, J.Crew, Macy's, Currys, Shopify, HSBC: a wall of shame of real Black Friday failures, what actually caused each one, and the MaxoPerf test YAML that finds it before your shoppers do.
- A year of load failures: what broke, why, and how to test for it · load testing, outages, performance testing, spike testing, capacity planning
Twenty real outages, ten failure patterns, one year. A guided tour of what load does to software in production, and how to test before it happens to you.
- ARC Raiders passed its pre-launch stress test — then overloaded anyway. Here's the soak lesson. · load testing, soak-growth-collapse, gaming
ARC Raiders ran clean for days after launch, then login queues and matchmaking broke as concurrent players kept climbing. Reporting attributes this to sustained growth beyond a one-time peak. Embark did not confirm a root cause.
- Battlefield 6 launch server queues — what 500,000 players waiting tells you about load testing · load testing, thundering-herd, gaming
On launch day EA queued over 500,000 concurrent players. Here's the thundering-herd pattern behind that decision and how to test for it before your next big release.
- AWS US-East-1 outage (October 2025): when recovery itself becomes the failure · load testing, retry-storm, cloud
A DNS race triggered the October 2025 AWS US-East-1 cascade. The recovery overload, with thousands of services reconnecting at once, turned a blip into 15 hours of disruption.
- Best Buy's Black Friday 2025 crash: what every retail team can learn about seasonal demand spikes · load testing, seasonal-demand-spike, e-commerce
Best Buy's site and app buckled under Black Friday morning traffic for roughly 90 minutes in November 2025. No official RCA was published, but the failure pattern is well understood.
- Clerk Cloud SQL migration outage — when a database operation becomes a load event · load testing, dependency-contention, saas
A routine storage migration turned into a 26-minute outage for Clerk in March 2026. The dependency-contention failure pattern it exposed applies to any team running a live database.
- Cloudflare outage (November 2025): the hard size limit that crashed a global proxy · load testing, config-limit-cliff, cloud
A database query change caused an oversized configuration file to exceed a hard internal limit, crashing Cloudflare's core proxy globally for ~3 hours in November 2025. The lesson is boundary testing before shipping config changes.
- Fiserv/Zelle banking outage 2025 — when a shared platform goes down, everyone goes down · load testing, dependency-contention, fintech
In 2025, a change at a Fiserv data center cascaded across a shared core-banking platform, taking Zelle and 60+ banking apps offline for over 12 hours during a payday window.
- GitHub capacity outages 2025–2026 — what a year of failures teaches about load testing · load testing, capacity-ceiling, saas
GitHub suffered dozens of capacity-related outages between 2025 and early 2026. Here's what caused them and how to test for the same failure pattern.
- Hollow Knight Silksong crashed Steam and three storefronts — the scheduled-release spike every publisher should test for · load testing, scheduled-release-spike, gaming
Silksong's no-pre-orders launch decision concentrated 535,000 day-one players into a single minute and brought down Steam, PS Store, and eShop. Here's how to test for it.
- Google Cloud outage (June 2025): the retry flood that extended a 40-minute fix into 3 hours · load testing, retry-storm, cloud
A null-pointer bug in new quota-policy code crash-looped Google Cloud's request-authorization service globally in June 2025. Most regions recovered in 40 minutes, but us-central1 spent 3 hours fighting a retry flood caused by missing backoff logic.
- IRS "Where's My Refund?" went down during peak filing season — what every team can learn · load testing, seasonal-demand-spike, government
In February 2026, the IRS refund tracker went offline in the middle of the filing surge. The lesson for every team: plan maintenance windows around load-tested seasonal peaks.
- NEET UG 2025 results day crash — what 2 million simultaneous logins look like · load testing, seasonal-demand-spike, education
On results day, 2M+ candidates hit neet.nta.nic.in at once, triggering 503/504 errors and blank scorecards. Here is what happened and how to test for it.
- Netflix NFL Christmas streaming issues — the live-event load problem with no time-shift · load testing, live-event-peak, streaming
Netflix viewers reported buffering and casting failures during NFL Christmas 2025. No confirmed RCA, but the live-event load pattern has a clear lesson.
- The Odyssey IMAX ticket onsale crash: what flash-sale concurrency really looks like · load testing, flash-sale-concurrency, e-commerce
When Nolan's The Odyssey went on sale, AMC and Fandango buckled under simultaneous demand for limited 70mm seats. Here's the failure pattern and how to test for it.
- OpenAI ChatGPT outage June 2025 — an illustrative AI-capacity story · load testing, capacity-ceiling, ai
A 10-plus-hour ChatGPT outage in June 2025 affected free, Plus, API, and Sora users. No official root-cause analysis was published. Here is what the pattern suggests and how to test for it.
- Shopify's Cyber Monday 2025 outage: what a shared auth chokepoint looks like under peak merchant load · load testing, thundering-herd, e-commerce
On Cyber Monday 2025, Shopify's admin, POS, and API access went down for hours while merchant storefronts largely survived. The company officially attributed the issue to a login authentication flow problem.
- Slack outage May 2025 — when infrastructure growth outpaces static configuration · load testing, capacity-ceiling, saas
In May 2025, Slack's growth hit a silent limit in its routing configuration, cutting live connectivity for users for nearly two hours. This post explains what the pattern means for any growing team.
- SSA "My Social Security" portal crashes — the cost of skipping a load test before launch · load testing, thundering-herd, government
In early 2025, the Social Security Administration portal crashed repeatedly after a software update. Reporting attributed it to staff not testing the software against high user volumes before launch.
- Stranger Things S5 premiere outage — when 30% extra capacity still isn't enough · load testing, scheduled-release-spike, streaming
Netflix pre-scaled ~30% for the Stranger Things S5 premiere but still went down for ~20 minutes. The spike-test lesson for every streaming team.
- UK banks payday outages 2025 — when month-end peak meets a fragile shared dependency · load testing, seasonal-demand-spike, fintech
In 2025, Lloyds, Halifax, TSB, Nationwide, and others failed on payday Fridays. The formal cause was not disclosed, but the pattern points to peak transaction volume colliding with a suspected shared dependency.
- API performance testing as a release gate · api performance, ci cd, release gates
A practical pattern for turning API load tests into CI/CD gates without hiding failures behind one average number.
- When private load generation beats public runners · byoc, private locations, load generation
Use private runner locations when the target is internal, regulated, latency-sensitive, or protected by network boundaries.
- Pair load tests with live observability · observability, run results, latency
What to watch during a load test: throughput, latency percentiles, errors, runner health, and the logs that explain failures.
- Why teams move load testing to the cloud · cloud load testing, managed runners, results
How managed runners, private locations, repeatable bundles, and shared run results make load testing easier to trust.
