AWS US-East-1 outage (October 2025): when recovery itself becomes the failure

On 19–20 October 2025, a significant portion of AWS US-East-1 services went down for roughly 15 hours. The trigger was a DNS race condition — a relatively contained event. What turned it into a prolonged, multi-service cascade was what happened next: the recovery itself became the failure.

This post examines how congestive collapse works, why recovery paths are as important to test as steady-state paths, and how to replicate the “everything reconnects at once” scenario in a MaxoPerf stress test before it happens to you.

What happened

In the early hours of 19 October 2025, a DNS race condition caused an internal endpoint table to be cleared, cutting off a core fleet-management service from its dependencies. Services that depended on that coordination layer began to fail.

So far, a serious but bounded event. The problem was that as engineers worked to restore the coordination service, every compute instance that had lost its lease tried to re-establish it at the same time. The control plane — suddenly receiving tens of thousands of simultaneous re-registration requests — became saturated. That saturation is what kept the outage running for 15 hours instead of 15 minutes.

The timeline

• 19 Oct, ~02:00 UTC — DNS race empties endpoint table; fleet-management service loses connectivity; downstream AWS services begin reporting errors. (AWS postmortem)
• ~02:30 UTC — Engineers identify the root cause and begin restoring the coordination service.
• ~03:00–17:00 UTC — Recovery attempts repeatedly stall as the control plane is overwhelmed by simultaneous lease re-establishment from thousands of instances — a classic congestive collapse.
• Mitigation steps — Engineers throttled work queues, restarted affected hosts in batches, throttled dependent services, and disabled specific health-check failover behaviors to reduce the reconnection storm.
• ~17:00 UTC — Services broadly restored after ~15 hours.

Why it happened

The initial DNS race was the trigger, but it would likely have caused a short-lived disruption on its own. The prolonged outage was a congestive collapse: a failure mode where mass simultaneous reconnection overwhelms the very system needed to complete recovery.

The missing defenses were:

• No randomized reconnection delay. All instances attempted to re-register within the same window. If each had waited a random jittered interval before retrying, the burst would have been spread into a manageable stream.
• No admission control on the recovery path. The control plane accepted every inbound reconnection attempt at once, rather than queuing or rate-limiting them.
• No load-shedding strategy. When the control plane became saturated, it had no mechanism to reject the lowest-priority work and protect capacity for the highest-priority re-registrations.

The engineers’ eventual mitigation — batching restarts, throttling work queues — was a manual approximation of the admission-control logic the system should have had built in.

The failure pattern

This incident is a textbook retry storm / congestive collapse. It generalizes far beyond AWS: any system that coordinates state across a large fleet faces this risk. The pattern appears whenever:

1. A dependency failure causes many clients to simultaneously lose their connection or session.
2. Those clients have no randomized backoff — they all retry at once.
3. The recovery path is not rate-limited or protected — it absorbs the full burst.

You don’t need to run an AWS-scale fleet to encounter this. A single microservice losing its connection to a database, a set of worker processes all losing a queue connection, or a mobile app with millions of users experiencing a brief auth outage — all of these can reproduce the same dynamics at smaller scale.

How it could have been prevented

The engineering literature on this is well established. The defenses are:

Jittered exponential backoff. Clients that lose a connection should wait before retrying, and the wait should include a random component so retries are distributed across time rather than synchronized. This alone breaks the burst into a manageable stream.

Admission control on recovery paths. Control planes and coordination services should be able to queue or rate-limit inbound reconnection requests. Allowing unbounded concurrent reconnects is the direct cause of congestive collapse.

Load shedding. When a system is saturated, it should reject or deprioritize lower-priority work so the most critical operations can proceed. A reconnection storm is a good candidate for tiered priority — re-register the services that other services depend on first.

Batch restarts. AWS’s eventual mitigation was batching instance restarts. This is a manual load-shedding strategy. Building it into the system design (vs. applying it during an incident) shortens recovery from hours to minutes.

Pre-tested recovery paths. The most reliable way to know your recovery path holds under reconnection load is to test it before the incident.

How to test for this with MaxoPerf

The goal is to simulate a “mass simultaneous reconnect” scenario on your own services and observe whether recovery degrades gracefully or collapses.

Engine: k6 or Taurus (JMeter).

Workload model: Open model — you want to control the arrival rate independently of service response time, since real reconnect storms are time-driven, not capacity-driven.

Profile (example):

1. Baseline phase — 50 virtual users for 5 minutes, confirming a clean steady state.
2. Disruption simulation — drop to 0 VUs for 30 seconds (simulating the outage window during which connections are severed).
3. Reconnect burst — immediately ramp to 500 VUs within 10 seconds and hold for 5 minutes. This is your “everything reconnects at once” moment.
4. Observe — does throughput recover, or does it spiral downward (congestive collapse)?

Target: Your own staging environment’s session-management, lease-renewal, or coordination endpoint — whatever path clients traverse when re-establishing state after a disruption.

Execution locations: Run from ≥2 managed MaxoPerf regions simultaneously to represent geographically distributed clients all reconnecting at once.

What to read in MaxoPerf results:

• Error rate at the reconnect burst — does it spike and then recover, or does it stay elevated?
• p95/p99 latency during the burst — a large jump that never comes back down signals saturation.
• Throughput plateau — if successful reconnects per second plateaus far below the inbound rate, the coordination service is backed up.

Run this scenario twice: once without backoff logic in your client, and once with jittered backoff enabled. The difference in error rate and recovery time is the proof point for your incident-response playbook.

For CI integration, consider running a lighter version of this recovery stress test — shorter burst window, lower VU count — as a performance gate on every release that touches your reconnection or session code.

Key takeaways

• The AWS October 2025 outage lasted 15 hours not because the initial DNS fault was severe, but because the recovery path had no protection against mass simultaneous reconnection.
• Congestive collapse is a predictable failure mode. Any system where many clients share a connection to a coordination layer is at risk.
• Jittered backoff is non-negotiable. Synchronized retries turn recoverable faults into extended outages.
• Test the recovery path, not just steady state. A service that performs well under normal load may collapse when asked to absorb a reconnection burst — and you won’t know until you test it.
• Admission control belongs in the system, not in the incident runbook. Batching restarts manually during an incident is the expensive way to learn this lesson.