Soak and stability for multi-hour sales

A system that survives a 10-minute spike test may still fall over at hour 6 of Cyber Monday. The failure mode is different: not connection pool exhaustion under sudden load, but gradual resource depletion over time. Memory that is not freed accumulates. File descriptors that are not closed run out. Database connection pools that do not evict stale connections fill up. Query plans that were fast when the table had 10,000 rows become slow when it has 50 million.

Soak tests are the only way to find these problems before they find your customers.

Before you start

• The full-funnel load test from end-to-end journey load passes cleanly at your modeled peak VU count.
• The spike and stress tests from spike and stress for sales show at least 1.5× headroom above your modeled peak.
• The target environment will not be needed for other work during the soak window — typically 6–12 hours.
• Application-level metrics (JVM heap, process RSS, open file descriptors, DB connection count) are being captured alongside the MaxoPerf run so you can correlate.

The BFCM stability problem

BFCM is not a 15-minute test. Black Friday sustained traffic runs for 18+ hours. Cyber Monday extends it further. Systems that pass every 15-minute load test but have a slow memory leak will OOM-crash at hour 8. Systems with connection pool settings tuned for daily traffic will exhaust their pool under sustained 3× load over hours, not minutes.

The soak test is not redundant with the load test. It is complementary — same load, much longer duration, looking for slow drift rather than peak values.

What to look for: the drift indicators

Unlike a load or stress test where you watch for peak metric values, a soak test is about trend over time:

Indicator	Healthy pattern	Concerning pattern
p95 latency	Flat across the full duration	Slow upward drift (>10 % over hours)
Throughput	Flat or very slight decline (<2 %)	Gradual throughput decline at constant VUs
Error rate	Near-zero throughout	Errors that appear only after 2–3 hours
Heap / RSS (app metric)	Stable or GC-bounded oscillation	Monotonically increasing — no ceiling
DB connection count	Stable pool usage	Gradually reaching pool limit
Open file descriptors	Stable	Growing without bound

A 10 % p95 drift over 8 hours is a signal. A 40 % drift or errors appearing in the final 2 hours requires immediate investigation — something is leaking.

Soak test configuration

Use the same VU count as your modeled peak load. Do not increase VUs to “make it more interesting” — that is a stress test. The soak test is asking: does the system stay stable at its operating point for the duration of a real BFCM sale?

execution:
  - scenario: bfcm-soak
    concurrency: 1360
    ramp-up: 5m
    hold-for: 8h

scenarios:
  bfcm-soak:
    think-time: 3s
    requests:
      - url: https://api.staging.example.com/v1/homepage
        label: GET /homepage
        think-time: 4s
      - url: https://api.staging.example.com/v1/products/${PRODUCT_ID}
        label: GET /product/:id
        think-time: 3s
      - url: https://api.staging.example.com/v1/cart/items
        label: POST /cart/items
        method: POST
        headers:
          Content-Type: application/json
          Authorization: Bearer ${SESSION_TOKEN}
        body: '{"product_id": "${PRODUCT_ID}", "qty": 1}'
        think-time: 5s
      - url: https://api.staging.example.com/v1/checkout/start
        label: POST /checkout/start
        method: POST
        headers:
          Authorization: Bearer ${SESSION_TOKEN}

Name the test bfcm-2025-soak-8h and schedule it to start in the evening so the result is ready for review the next morning. See Schedule recurring tests.

Set failure criteria with generous thresholds — the soak test should auto-fail only on serious degradation:

• p95 latency (any label) > 3000ms → fail
• error rate > 2 % → fail

Tighter criteria are useful for regression detection but not for an exploratory soak that is looking for drift patterns.

Running the soak in MaxoPerf

1. Upload the YAML and configure the test in the console as described above.
2. Use the Schedules tab to fire the run at a predetermined time — typically 22:00 the night before you need the result.
3. Do not watch the run live for 8 hours. MaxoPerf streams metrics continuously and the full 8-hour timeline is available for review when the run completes.
4. The morning after, open the Overview tab and focus on the full time axis — not the current window. Use the time range selector to view the entire run duration.

Correlating soak results with application metrics

The soak run result tells you that something is leaking. Application metrics tell you what. When reviewing a soak result that shows drift:

1. Note the time at which drift becomes visible in the p95 chart (e.g., “drift begins at t+4h”).

2. Open your APM or metrics dashboard and look at the same time window for: heap growth, GC frequency, DB connection pool usage, open file descriptor count.

3. The metric that trends upward in the same time window as p95 drift is the root cause.

4. Common causes and fixes:

   Root cause	Symptom	Fix direction
   Memory leak in app code	Heap grows continuously; GC pauses lengthen	Fix object retention; heap dump analysis
   Connection pool not evicting	DB connections approach pool limit	Set idle eviction timeout and max-lifetime
   File descriptor leak	FD count grows; eventually EMFILE errors	Close resources in finally blocks; audit stream handling
   Query plan regression under table growth	p95 for DB-backed endpoints drifts	Add index; ANALYZE/VACUUM; pagination
   Log rotation bloat	Disk fills; latency spikes from blocked writes	Rotate logs more frequently; async log writing

Multi-hour soak for the dress rehearsal

At T-1 week, your dress rehearsal soak should match the expected real-event duration:

• Black Friday start to Cyber Monday end — 4 days, but practically: test the longest continuous high-traffic window, which is typically 18–24 hours.
• A 24-hour soak at modeled peak VUs is the gold standard for a BFCM dress rehearsal.
• If runner budget limits a 24-hour run, a 12-hour soak with clean results (no drift) is an acceptable proxy.

Do / don’t

Do	Don’t
Use the same VU count as your modeled peak	Increase VUs during the soak — that changes what you’re testing
Schedule the soak to run overnight and review in the morning	Sit watching a live dashboard for 8 hours
Correlate soak drift with application-level metrics	Dismiss gradual drift as “noise”
Run the soak after the stress test passes	Run a soak before establishing that the system handles peak load at all
Set failure criteria — even if generous — so the run auto-fails on collapse	Rely on visual inspection of an 8-hour chart to detect a failure

Where to go next

• Soak / endurance test — the full soak test reference.
• Third-party and payment dependencies — include dependency call patterns in the soak scenario.
• War-room and runbook — use soak results to set drift thresholds for game-day alerting.
• BFCM readiness checklist — the soak is one of the required checklist items before go/no-go.