Cloudflare outage (November 2025): the hard size limit that crashed a global proxy

On 18 November 2025, Cloudflare’s core proxy service crashed globally, taking down traffic for a significant portion of internet users for roughly three hours. The root cause was not a traffic surge. It was a database query change that produced a configuration file too large for the proxy to handle — and the proxy had no way to handle it gracefully.

This is the “config-limit cliff” failure pattern: a hard limit inside a system, unknown or unguarded, that causes a complete crash rather than a partial degradation when input exceeds it. The lesson is about boundary testing — specifically, testing what happens when configuration data approaches and exceeds its limits — before that configuration ships to production.

What happened

In the weeks before 18 November, Cloudflare’s Bot Management feature used a configuration file to define its active detection rules and feature flags. That file was generated by querying a database that tracked which features were enabled.

A database permissions change altered how that query ran. Without a filter that had been implicitly enforced by the previous permission set, the query returned duplicate rows. The result: the generated configuration file contained more than twice its normal number of entries, pushing it well above 200 items.

The core proxy that loaded this configuration file had a hard limit of 200 items. When it encountered a file that exceeded the limit, it did not fall back to the previous valid configuration. It did not log a warning and skip the oversized file. It panicked — an unhandled error in the proxy’s configuration-loading code — and crashed.

Because this crash occurred globally, across all Cloudflare data centers, the proxy was unavailable worldwide for the duration.

The timeline

• 18 Nov, before 11:28 UTC — A database permissions change is deployed; the configuration query begins returning duplicate rows in the background.
• 11:28 UTC — A scheduled configuration refresh causes the oversized Bot Management configuration file to be distributed to proxy instances globally. Proxies begin crashing as they attempt to load the file.
• ~11:30–14:30 UTC — Core proxy is unavailable globally (~3 hours of widespread disruption). Cloudflare engineers identify the oversized configuration file as the cause.
• Mitigation — Engineers push a corrected configuration file that stays within the 200-item limit. Proxies reload and recover.
• ~14:30 UTC — Service restored for most users. Full recovery confirmed by ~17:30 UTC.

Why it happened

The trigger was a database permissions change that removed an implicit query constraint, causing duplicate entries to be returned. But the cascading failure required a second condition: the proxy’s configuration loader had no graceful handling for inputs that exceeded its hard size limit.

Cloudflare’s postmortem identifies both problems:

1. The query change was not validated for its effect on output size or item count before deployment.
2. The proxy’s configuration-loading code treated an oversized input as a fatal, unrecoverable error — a panic — rather than rejecting the bad input and continuing to operate with the last known-good configuration.

Neither condition alone would have caused a global outage. Together, they produced one. The database change was the “trigger.” The hard limit with no graceful degradation was the “cliff.”

The failure pattern

This is a config-limit cliff — a category of failure where a hard boundary exists inside a system, is not tested at deployment time, and causes a crash rather than a degradation when crossed.

Config-limit cliffs are especially dangerous in globally distributed infrastructure because:

• A single configuration deployment reaches all instances simultaneously. There is no canary window.
• If the failure mode is a crash rather than a rejection, the entire fleet fails at once.
• The fix requires generating and deploying a corrected configuration before any instance can recover — there is no self-healing.

The pattern generalizes to any system that accepts structured configuration input: request routers, feature-flag systems, WAF rule sets, API gateway configurations, serialization schemas. Whenever a “configuration” entity has a hard limit on size, count, or depth, and the system does not test inputs against that limit before accepting them, a config-limit cliff exists.

How it could have been prevented

Validate configuration output size before deployment. The query change that doubled the configuration file size should have been caught by a pre-deployment check: “does this query return more rows than the configuration consumer can handle?” Configuration generation pipelines should include size assertions, not just syntax checks.

Fail open, not closed, on oversized configuration. When the proxy encountered a configuration file that exceeded its limit, the correct behavior was to reject the new file, log an error, and continue operating with the previous valid configuration. A panic that crashes the process is the worst possible response to a bad input.

Apply the same limit in the generator. The system that generates and distributes configuration files should enforce the same 200-item limit that the consumer enforces, refusing to distribute a file that would violate it.

Canary configuration deployments. Rolling out configuration changes to a small subset of instances first would have surfaced the crash before it reached every data center.

Boundary tests in the pipeline. Automated tests that produce files at exactly the limit, one below, and one above — verifying the consumer’s behavior in each case — would have caught both the missing size check and the panic behavior.

How to test for this with MaxoPerf

The configuration-limit cliff pattern calls for two complementary test types: a configuration test that validates behavior at and around the hard limit, and a volume test that confirms the system handles large-scale input variation without silent failures.

Engine: k6 or Taurus.

Workload model: Closed model — you are testing correctness at specific input boundaries, not raw throughput.

Profile (example):

Phase 1 — Boundary test: Send requests producing payloads at 80%, 100%, and 110% of the documented limit. At 100%, the system should accept and respond correctly. At 110%, expect a clean 4xx rejection — not a crash or 5xx. Use 10–20 VUs, 2 minutes per phase.

Phase 2 — Volume test under configuration churn: Simulate 1 configuration update per second for 10 minutes while baseline load traffic continues at 200 VUs. Verify that reloads — even large ones — do not spike the error rate.

Target: Your own staging configuration-loading, feature-flag evaluation, or rule-set ingestion endpoint.

Execution locations: A single managed MaxoPerf region for boundary testing; ≥2 regions for distributed configuration delivery volume tests.

What to read in results:

• Error rate at 110% boundary — clean 4xx, not 5xx or timeout.
• Error rate during churn — near zero; any spike signals a gap in reload isolation.
• p99 latency during reloads — a large spike suggests the reload blocks request handling.

Use the run artifacts tab to confirm rejection messages are structured responses, not crash traces. Run the boundary test as a CI gate on any deployment that touches configuration generation or consumption.

Key takeaways

• The November 2025 Cloudflare outage was caused not by traffic volume but by a configuration file that exceeded a hard internal limit with no graceful fallback.
• A hard limit with no graceful degradation is a cliff. Any system that panics or crashes on oversized input is one bad deployment away from a global outage.
• Configuration generation and configuration consumption must share the same size constraints. If the consumer enforces a limit, the generator must enforce the same limit before distributing.
• Boundary testing belongs in the deployment pipeline. Testing configuration inputs at, below, and above their limits — before any deployment — catches these failures in staging rather than in production.
• Fail open on bad configuration. Reject the new file; continue with the last known-good state. This converts a potential outage into a logged error.