Clerk Cloud SQL migration outage — when a database operation becomes a load event

Authentication and identity services sit on the critical path of almost every user action in a modern application. When they degrade, nothing else works. On March 10, 2026, Clerk — a developer authentication and user management platform — experienced a 26-minute outage traced directly to a routine database storage operation that interacted badly with live production load.

The failure is a precise example of a class that catches many engineering teams off-guard: the dependency-contention pattern, where a degraded or temporarily slow dependency cascades into service-wide saturation.

What happened

According to Clerk’s own postmortem, a Cloud SQL live migration ran on the production database without prior customer announcement. The migration elevated disk latency. That elevated latency propagated upward: database operations that normally completed quickly now held locks longer, which increased lock contention across the connection pool. As contention grew, compute saturated. The system could not process incoming authentication requests fast enough, and the overflow surfaced to callers as HTTP 429 responses.

The outage lasted approximately 26 minutes and resolved on its own when the migration finished. It was the second similar incident in seven months. Following the event, Clerk announced a move to maintenance windows using replica promotion rather than live migrations.

The timeline

• March 10, 2026, onset: Cloud SQL live migration begins on the production database without advance user notice.
• Elevated disk latency: Storage I/O takes longer than normal; database operations queue while waiting for disk responses.
• Lock contention rises: Slower operations hold shared resources longer, blocking other queries; the contention compounds.
• Compute saturation: The connection pool fills; worker threads are occupied waiting; new requests cannot be admitted.
• 429 responses: Incoming authentication requests begin returning 429 errors as the service sheds load it cannot process.
• ~26 minutes in: Migration completes; disk latency returns to normal; contention clears; service recovers without manual intervention.

Why it happened

The direct cause was a scheduling choice: a live migration ran during production hours on the live database. But the underlying vulnerability was already present before that decision — the service had no buffer between a degraded dependency and user-visible failure.

When storage latency increased, every database operation that touched disk slowed down. Those slower operations held row locks and connection-pool slots for longer than expected. Once enough operations were stacked up waiting, new requests could not acquire what they needed. The system did not have a mechanism to queue gracefully or prioritize critical paths; it simply saturated and returned errors.

This is dependency-contention: a normally healthy dependency becomes temporarily slower, and the service has no resilience layer to absorb the degraded response. The failure cascades from the database layer up through compute and out to the API surface.

The failure pattern

Dependency-contention failures share a common shape: the service works normally up to a threshold of dependency latency, then degrades rapidly once the dependency crosses into a range where lock and connection exhaustion begin. The nonlinearity is what surprises teams — the first 20ms of additional database latency is invisible to users; the next 20ms sends error rates to 100%.

What makes this pattern particularly relevant for any team running a live database is that the trigger does not need to be a migration. The same cascade can happen during a slow backup, an index rebuild, a noisy neighbor on shared infrastructure, or any maintenance operation that temporarily degrades storage.

How it could have been prevented

Schedule maintenance operations away from production load. Migrations, index rebuilds, and other storage-intensive operations should be planned during low-traffic windows, or run against a promoted replica with a traffic cut-over, not against the live primary while full traffic flows through it.

Test the degraded-dependency case before any migration. Before running a migration in production, validate in staging how the service behaves when database latency is elevated. If the test reveals that 50ms of extra disk latency causes connection-pool exhaustion, that is a critical signal to fix before the real migration runs.

Add circuit-breaking and queue depth limits. A service that can detect a degraded dependency and begin shedding non-critical work — or queuing it with a short timeout — converts a 26-minute outage into a brief degradation for some request types rather than total unavailability.

Track this class of risk against your error budget. A 26-minute outage for an authentication service burns a substantial portion of a monthly error budget. Migrations that carry non-trivial risk should be gated on whether the error budget has sufficient headroom, and that headroom should be visible before the change is scheduled.

How to test for this with MaxoPerf

The goal is to observe your own service’s behavior when its database dependency degrades — before that degradation happens in production under real load.

Set up a staging environment with controllable storage latency. Most cloud database services have a test environment option. You can also introduce artificial latency at the network layer or use database-proxy tools in staging. The point is to have an environment where you control how long disk operations take.

Run a stress test against your staging authentication or primary API path. Configure a k6 or Taurus script with MaxoPerf targeting the endpoint that most depends on the database. Use an open workload model so that the number of in-flight requests does not throttle automatically as latency climbs. A representative profile might be:

• Ramp from 0 to your expected peak over 2 minutes.
• Hold at peak for 5 minutes with normal database latency.
• Introduce elevated storage latency (simulate the migration condition — add 50ms, then 100ms, then 200ms).
• Hold for 5 minutes at each latency level.
• Return to normal latency and observe recovery.

Watch these signals in MaxoPerf results:

• Error rate — the key indicator; note the latency level where it departs from baseline.
• p95/p99 response time — climbs before errors do; the point where p99 decouples from p95 is often where lock contention is beginning to cascade.
• Throughput — successful requests per second will plateau or drop before error rate spikes; this is the connection-pool exhaustion signal.

Record the threshold and build a pre-migration checklist. Once you know that your service starts returning errors at X ms of database latency, you have a concrete number to require from your migration plan: “this migration must not elevate disk latency above X ms for more than Y seconds.” Run the stress test again after adding circuit-breaking or connection-pool limits to verify the fix actually works.

If your test or production environment is not reachable from the public internet, use MaxoPerf’s private/BYOC execution locations to run the same stress profile from inside your network without changing the test or results workflow.

Key takeaways

• A healthy service can be made unavailable by a healthy database operating slower than usual. The connection between dependency latency and service error rate is often nonlinear — the dangerous zone is reached quickly.
• Live migrations are load events. Treating them as transparent background operations ignores the reality that disk-latency spikes propagate upward into lock contention and compute saturation.
• Test the degraded-dependency case, not just the happy path. A stress test with elevated database latency reveals the cascade point before a migration reveals it to users.
• Circuit-breaking and queue limits convert cascades into partial degradations. A service that can detect contention and limit in-flight requests survives a slow dependency; one without that protection does not.
• Error budget awareness prevents surprise. If you know your monthly budget and how much a 26-minute authentication outage costs, you can make informed decisions about when and how to run high-risk maintenance.