Google Cloud outage (June 2025): the retry flood that extended a 40-minute fix into 3 hours

On 12 June 2025, Google Cloud’s request-authorization service — the layer that validates quotas and policies for API calls across the platform — crash-looped globally. The underlying bug, a null-pointer error introduced in a quota-policy code change, was rolled back within about 40 minutes for most of the world.

In us-central1, the outage ran for roughly three hours. Not because the bug was harder to fix there. Because the clients that had been failing during the outage began retrying all at once, without backoff, and the recovering authorization datastore could not absorb the burst.

This post looks at how missing backoff logic converts a short incident into a prolonged one — and how to test whether your own services are resilient to this pattern.

What happened

On the morning of 12 June 2025, Google Cloud engineers deployed a change to the quota-policy evaluation path inside their request-authorization service. The new code introduced a null-pointer error that was triggered when it encountered policy entries with blank fields — a condition that the new policy deployment immediately produced.

The authorization service began crash-looping globally. Because nearly every Google Cloud API call passes through request authorization, the effect was broad: Compute, Storage, Kubernetes, and many other services began returning errors to customers. The outage affected services across multiple regions simultaneously.

Engineers triggered a “red-button” rollback — reverting the problematic code change — and most regions stabilized within approximately 40 minutes.

The timeline

• 12 Jun, ~09:45 UTC — Quota-policy change is applied; new policy entries include blank fields. Authorization service begins crash-looping globally.
• ~10:00–10:30 UTC — Customer-facing errors spread across Compute, Storage, GKE, and other services. Error rates climb sharply.
• ~10:25 UTC — Google engineers trigger a red-button rollback of the problematic code change.
• ~10:30–10:45 UTC — Most regions stabilize as the rollback takes effect. Authorization service recovers in the majority of Google Cloud.
• ~10:45–13:00 UTC — us-central1 remains degraded. The authorization datastore serving that region is overwhelmed by retry traffic from clients that had been failing during the crash-loop — clients retrying without randomized exponential backoff, flooding the recovering service faster than it can process requests.
• ~13:00 UTC — us-central1 authorization service fully recovers after ~3 hours of degradation.

Why it happened

The initial crash-loop had a clear, fixable cause: a null-pointer error in new code, triggered by a policy with blank fields. That is the kind of bug code review and integration testing should catch before deployment.

The us-central1 extension had a different cause: retry behavior without randomized exponential backoff. When the authorization service crashed, every client began receiving errors and retrying — correctly. But without a randomized wait before each retry, all clients sent their next request at nearly the same moment.

The authorization datastore was asked to absorb the full accumulated demand of every failing client simultaneously, at the exact moment it was trying to recover. It became overloaded, causing more failures, triggering more retries, sustaining the loop. The two components were independent — together they compounded a 40-minute fix into a 3-hour regional event.

The failure pattern

This is a retry storm — one of the most reliable ways to extend an outage. The pattern appears whenever:

1. A shared service becomes temporarily unavailable.
2. A large number of clients simultaneously begin retrying.
3. The retries are not spread over time (no jitter, no backoff, or insufficient backoff).
4. The recovering service absorbs the full burst of accumulated retries, becomes overloaded again, and fails to complete its recovery.

The retry storm is the cloud-era thundering herd: accumulated failed requests all re-arriving at once. The more clients there are and the longer the outage runs, the larger the burst when the service begins to recover.

Every SLO that promises fast recovery implicitly assumes that client retry behavior does not prevent recovery. That assumption is worth testing.

How it could have been prevented

Jittered exponential backoff in every client. When a call fails, the client should wait before retrying, with a random component (jitter) so that many clients do not all retry at the same moment. With jitter, the accumulated retry burst spreads into a recoverable stream.

Admission control on the recovering service. Even with client-side backoff, a recovering service should accept work at a controlled rate — queuing or rejecting excess requests — until it stabilizes.

Circuit breakers. Client-side circuit breakers stop retries entirely when error rates exceed a threshold, reducing recovery traffic to a trickle rather than a burst.

Pre-deployment policy validation. The null-pointer bug was preventable: a test applying the new code to a policy entry with blank fields would have surfaced the crash before deployment. Configuration changes to shared authorization services deserve the same rigor as code changes.

How to test for this with MaxoPerf

The test goal is to simulate what happens to your service when it recovers from a brief outage and is immediately hit by a retry burst from all the clients that were failing.

Engine: k6 or Taurus (JMeter).

Workload model: Open model — you want to control the arrival rate independently of service response time, because retry storms are rate-driven by the number of queued-up clients, not by server capacity.

Profile (example):

Phase 1 — Steady state: 200 virtual users for 5 minutes. Establish clean baseline p95/p99 and near-zero error rate.

Phase 2 — Simulated outage: Return 503s for 2 minutes. Virtual users continue issuing requests; all fail. This is the accumulation window.

Phase 3 — Recovery burst: Restore the endpoint. Ramp from 0 to 600 virtual users within 5 seconds — all clients retrying at once. Hold 5 minutes and observe.

Phase 4 — Control run with backoff: Repeat but configure the client with jittered exponential backoff (e.g., 0.5–1.5s initial delay, doubling, capped at 30s). Compare error rate and recovery time against Phase 3.

Target: Your own staging authorization, session-validation, or request-routing endpoint — wherever clients accumulate retries when a shared service is unavailable.

Execution locations: ≥2 managed MaxoPerf regions to model distributed clients resuming retries simultaneously.

What to read in results:

• Error rate in Phase 3 — recovers quickly with backoff; stays elevated (congestive collapse) without it.
• p99 latency — returns to baseline within seconds with good backoff; stays high without it.
• Throughput plateau — successful req/s far below inbound rate means the service is backed up.

Compare the no-backoff and jittered-backoff runs in the MaxoPerf results view. The delta in recovery time measures your retry-storm resilience. Include this as a release gate on anything touching retry configuration or shared service infrastructure.

Key takeaways

• The June 2025 Google Cloud outage had two independent causes: a null-pointer code bug, and missing backoff logic in clients. The first was fixed in 40 minutes; the second kept us-central1 down for 3 hours.
• Retry storms are predictable and testable. The behavior of clients retrying without backoff is deterministic: they all retry at once, and the recovering service can’t handle it.
• Jittered exponential backoff is not optional. Any client that calls a shared service and retries on failure must add randomized delay. This is the foundational defense against retry storms.
• Your SLO recovery time objective is only achievable if your retry behavior cooperates. Test whether your clients’ retry patterns protect or undermine recovery — they often do the latter.
• Stress test the recovery path, not just the happy path. A service that performs well at steady state may still fail to recover under a retry burst. Only a stress test modeled on realistic client retry behavior reveals this.