Slack outage May 2025 — when infrastructure growth outpaces static configuration

Most capacity failures are visible in advance: traffic climbs, latency increases, and engineers have some signal that the system is approaching a limit. A subtler class of failure occurs when the limit is not in the application code itself but in a static configuration that was correct when it was set and has quietly become wrong as the system grew.

In May 2025, Slack experienced this pattern directly. The details reported by postmortem aggregators describe the incident with enough clarity to illustrate how infrastructure growth can create invisible ceilings — and why load testing at realistic scale is the most reliable way to find them.

Note: this incident occurred in May 2025, just before this review’s twelve-month window. It is included here because the failure pattern is directly relevant to any team running a growing SaaS platform, and because the underlying dynamic is ongoing rather than historical.

What happened

On May 12, 2025, Slack experienced a disruption lasting approximately one hour and 58 minutes. According to postmortem reporting, the cause was described this way: “Infrastructure growth had outpaced static configurations, preventing routing updates from reaching the web layer.” When routing updates could not propagate to the web layer, clients lost live connectivity data. Instead of receiving fresh routing information, they were working from stale state. The result was elevated errors and degraded connectivity for a significant portion of users.

The incident resolved once the routing configuration was updated to match the scale of the current infrastructure. No data was lost; the impact was connection and messaging disruption.

The timeline

• May 12, 2025, onset: Infrastructure operating at a scale that has grown beyond what static routing configuration can describe.
• Routing propagation fails: Routing updates cannot reach the web layer; the configuration ceiling is hit.
• Clients lose live routing data: Web clients receive stale or incomplete routing state; live connectivity degrades.
• Elevated errors: Users experience disconnections, message delivery failures, and degraded real-time features.
• ~1h58m later: Routing configuration is updated; propagation resumes; service recovers.

Why it happened

The direct cause was a static configuration that defined how routing information could be described and propagated, but that configuration had a size or count limit set when the infrastructure was smaller. As the infrastructure grew — more hosts, more routes, more services — the configuration remained the same. At some point, the infrastructure exceeded what the configuration could represent.

This is a confidence-medium incident: postmortem aggregators report the routing-configuration cause clearly, but the precise nature of the static limit is not publicly documented. The pattern is reliable; the exact technical boundaries are not all public. What is consistent is the core dynamic: a hard static limit, invisible during growth, becomes a ceiling that is only discovered when it is crossed.

The failure pattern

This is a capacity-ceiling failure of a specific sub-type: a silent configuration limit. Unlike an infrastructure capacity ceiling (where adding servers expands headroom), a silent configuration limit cannot be resolved by scaling compute. It requires identifying the static value, understanding why it exists, and updating it — work that cannot happen in real time once users are already affected.

Silent configuration limits tend to share a few properties that make them dangerous:

• They have no warning signal. There is no metric that says “you are at 80% of this limit.”
• They are often set once and forgotten, because they were set when the system was much smaller and the limit seemed generous.
• When they are crossed, the failure is often non-obvious because it does not look like a resource exhaustion problem — it looks like a routing or networking problem.

Any team running infrastructure that has grown significantly since its initial configuration should treat this pattern as an active risk.

How it could have been prevented

Audit static configurations for hidden limits. Routing tables, configuration files, and infrastructure manifests that define the shape or size of the system should be reviewed regularly as infrastructure grows. Limits that made sense at 100 nodes may be dangerously close at 1,000.

Test at projected fleet sizes, not current sizes. A scalability test that runs at 1.5× or 2× current infrastructure size does not just reveal compute limits — it also exercises the paths where routing and configuration data must describe a larger system. Configuration limits surface here.

Add limit-proximity alerting. Where configuration limits can be quantified (e.g., maximum route table size, maximum entry count), instrument monitoring to alert when the system reaches 70–80% of that limit. The goal is to discover the limit before the ceiling, not at it.

Define and track an SLO. An SLO converts abstract reliability goals into a measurable target. When you know your target — say, 99.9% monthly availability — you can calculate how many minutes of outage remain in your error budget. That quantification creates urgency around finding and fixing hidden limits before they consume the budget.

How to test for this with MaxoPerf

The goal is to find configuration ceilings before they find you by testing at the scale your infrastructure will reach, not just the scale it is at today.

Run a scalability test at projected fleet sizes. If your infrastructure is expected to double in the next two quarters, configure a scalability test that steps through 25%, 50%, 75%, 100%, 150%, and 200% of your target scale. Use MaxoPerf with k6 or Taurus to simulate the web-layer load that would accompany each fleet size. A concrete example profile:

• 5 virtual users for 2 minutes (baseline).
• Step to 25, 50, 100, 200, 400, and 800 VUs, holding each for 3 minutes.
• Measure p95 latency, error rate, and throughput at each step.

Watch for non-linear degradation. A healthy, well-configured system degrades roughly linearly — throughput climbs as VUs climb, latency stays stable until real saturation. A silent configuration limit shows up differently: the system is fine until a specific VU count, and then latency or errors jump suddenly, not gradually. That sudden jump is the limit.

Pair with a breakpoint test at your traffic growth horizon. A breakpoint test starting from today’s peak and stepping up to 3× or 5× current traffic will exercise routing and configuration paths at scales you have not run before. Note any sudden changes in behavior that do not correlate with compute metrics.

Use multiple managed regions to simulate realistic geographic traffic. Slack’s routing disruption involved how routing updates propagate to the web layer. Testing from a single point of origin may not surface the propagation failure that geography would reveal. MaxoPerf’s managed cloud locations let you generate load from multiple regions simultaneously, which is closer to the distribution a real production failure would need to survive.

Review run results carefully after each step increase — not just for latency and error rate, but for any change in behavior patterns that looks abrupt rather than gradual.

Key takeaways

• Static configuration limits are invisible until crossed. Unlike compute capacity, there is often no metric warning that a configuration ceiling is approaching.
• Growth creates the risk silently. The configuration that was safe at 100 nodes may be dangerous at 1,000. The limit does not change; the system does.
• Test at tomorrow’s scale, not today’s. Scalability and breakpoint tests at projected fleet sizes are the most reliable way to find configuration ceilings before users do.
• Non-linear degradation in a test is a signal. If throughput or error rate jumps suddenly at a specific load level rather than degrading gradually, a hard limit — not just resource saturation — is the likely cause.
• An SLO makes the risk concrete. Knowing your availability target transforms a vague concern about capacity into a quantified risk that can be prioritized alongside feature work.