War-room and runbook

import Screenshot from ‘@components/Screenshot.astro’;

All the load tests in the world do not matter if your team does not have a clear, practiced plan for game day. The war-room is not a metaphor — it is a literal coordination structure: who is watching what metric, what thresholds trigger escalation, and exactly what actions to take when something goes wrong at 00:07 on Black Friday morning.

The war-room document is written before the event, rehearsed during the dress rehearsal, and executed on game day. Writing it under pressure at midnight is too late.

The war-room structure

A BFCM war-room has three layers:

Layer	Who	What they watch	What they act on
Frontline monitoring	On-call engineer(s)	MaxoPerf live dashboard, error rate, p95 latency	Escalate if thresholds are breached
Incident command	Engineering lead	Frontline escalations, business impact	Authorize abort, coordinate response
Business liaison	Product/ops lead	Revenue metrics, customer support volume	Communicate decisions to business stakeholders

Setting up the MaxoPerf live dashboard

During game day, the MaxoPerf Active runs view gives you a live window into all running load tests and their metrics. For a BFCM war-room, configure the following before the event:

1. Pin the active-runs workspace view. Open the workspace in the MaxoPerf console and bookmark the active runs page. Every engineer in the war-room has this URL open.
2. Pre-create the BFCM test definitions. Every test you will run on game day — doorbuster monitoring, checkout journey validation, payment-path synthetic — is pre-configured with failure criteria. Nothing is configured from scratch under pressure.
3. Set up the comparison baseline. Pin last year’s dress-rehearsal run (or the T-1 week dress rehearsal) as the comparison baseline. When game day starts, every new run can be compared to the baseline in one click.

The monitoring assignment matrix

Assign specific metrics and thresholds to named individuals. “Everyone watches everything” means no one catches a problem fast enough.

Engineer	Primary metric	Amber threshold	Red threshold
SRE lead	Overall error rate	> 0.5 %	> 2 %
Backend engineer 1	p95 latency — checkout flow	> 600 ms	> 1200 ms
Backend engineer 2	p95 latency — payment endpoint	> 800 ms	> 2000 ms
Infra engineer	Database connection pool utilization	> 70 %	> 90 %
Frontend engineer	CDN cache hit rate	< 85 %	< 70 %
DBA	Slow query rate	> 5/min	> 20/min

Each engineer checks their metric every 5 minutes and reports status in the war-room channel: green / amber / red.

Abort criteria

Define the conditions under which you will take an action — reduce traffic, roll back, or abort the sale entirely — before the event. Making these decisions under pressure with revenue numbers in front of you leads to the wrong choices.

Condition	Duration	Action
Error rate > 2 %	Sustained 3 min	Amber: engineering escalation, investigate
Error rate > 5 %	Sustained 2 min	Red: incident command decision required
Error rate > 10 %	Any	Immediate rollback / traffic shed
p95 payment > 3000 ms	Sustained 5 min	Throttle checkout entry, open incident
Payment service 503s	Any	Circuit break to fallback; incident command
Database connection pool > 95 %	1 min	Immediate action; possible pod restart
Complete checkout unavailability	30 s	Activate maintenance page; full incident

Write these criteria in the runbook, reviewed and signed off by engineering lead and business lead at least one week before the event.

The runbook: action procedures

For each abort condition above, the runbook lists the exact commands or console actions to take. No one should be searching documentation at midnight on Black Friday.

Example runbook entry: high error rate (amber)

Condition: Overall error rate > 2 % for 3 minutes

Immediate actions (frontline engineer):

1. Open MaxoPerf run detail → Overview tab → Log/Errors tab.
2. Note the specific error type (HTTP 5xx? timeout? specific endpoint?).
3. Post in war-room channel: [AMBER] Error rate 2.4%. Endpoint: POST /checkout/payment. Error: upstream timeout. Investigating.
4. Check payment gateway dashboard for upstream issues.
5. Check infrastructure dashboard for pod health, CPU, memory.

Escalation: If error type is not identified within 5 minutes, escalate to incident command.

Resolution: When error rate drops below 1 % for 2 minutes, post: [GREEN] Error rate normalized. Cause: [description]. Monitoring continued.

Example runbook entry: complete checkout unavailability (red)

Condition: Checkout unavailable for > 30 seconds

Immediate actions (any engineer):

1. Post in war-room channel: [RED] Checkout DOWN. T=00:07. Activating maintenance page.
2. Execute maintenance page activation: kubectl apply -f deploy/maintenance-mode/checkout-maintenance.yaml
3. Notify incident command.
4. Begin incident response in dedicated incident channel.

Rehearsing the war-room

The dress rehearsal at T-1 week is not just a load test — it is a war-room exercise. Run the dress rehearsal with the full war-room structure active:

• All engineers in their monitoring seats
• The war-room channel open
• The runbook in hand
• Incident command role staffed

Deliberately introduce amber conditions during the rehearsal (e.g., artificially increase error rate on the mock payment server for 5 minutes) and practice the escalation flow. The goal is that on game day, every action is a repetition, not a first experience.

Post-event debrief

After the event, within 48 hours:

1. Run the year-over-year comparison in MaxoPerf: this year’s peak run vs. last year’s.
2. Document all amber and red events in the incident log.
3. Update the runbook with any actions that were taken but not documented.
4. Archive the game-day run in MaxoPerf with the tag bfcm-2025-game-day for next year’s baseline.

Where to go next

• BFCM readiness checklist — war-room setup is a required pre-event checklist item.
• Failure criteria pass/fail gates — the failure criteria that drive war-room alerts.
• Comparing runs and baselines — set up the year-over-year comparison during debrief.
• Daily countdown calendar — when in the T-6 week program the war-room runbook is written and rehearsed.