Failure Modes

1. The thundering herd: everyone at once

The market opens at 09:00. In that second:

• Every gateway pulls the symbol list from Redis.
• Every bridge connects to its provider.
• Every scheduled job fires on 0 0 9 * * *.
• Clients idle overnight reconnect.

None of these is heavy on its own. But all of them land in the same second. The system cannot handle in one second the load it carries comfortably for the rest of the day.

// 1) Scheduled job: offset by 0-30s instead of a fixed second
long offset = ThreadLocalRandom.current().nextLong(30_000);
scheduler.schedule(job, openTime + offset);

// 2) TTL: not fixed, ±20% spread
int ttl = 300;
int spread = ThreadLocalRandom.current().nextInt(-60, 61);
redis.setex(key, ttl + spread, value);

// 3) Staggered startup: wait based on pod index
int index = readPodIndex();            // StatefulSet ordinal or a hash
Thread.sleep(index * 500L);            // 0, 500, 1000, 1500 ms...

Ten lines in total. For us, peak Redis usage at market open dropped from 100% to 40% without adding capacity anywhere.

2. Cache stampede: the spread computed 200 times at once

A special and very common form of the herd. The symbol:EURUSD:spread key expires. At that instant 200 gateways read it, all 200 get a miss, and all 200 start computing it.

The computation is not expensive — but 200× is. And by the time it completes the key has expired again, so the loop feeds itself.

Three fixes

value = redis.get(key);
if (value != null) return value;

// Only one takes the lock
if (redis.set(key + ":lock", me, "NX", "PX", 5000)) {
    value = expensiveCompute();
    redis.setex(key, ttlWithSpread(), value);
    redis.del(key + ":lock");
    return value;
} else {
    // Others: serve the previous value (separate, longer-lived key)
    return redis.get(key + ":previous");
}

// Store, alongside the value, how long computing it took and when it expires.
// As expiry approaches, the PROBABILITY of refreshing rises.

long remaining = expiresAt - now();
double threshold = -computeMs * BETA * Math.log(Math.random());   // BETA ~ 1.0

if (remaining < threshold) {
    value = expensiveCompute();      // early, spontaneously, alone
    store(value);
}
return value;

3. Cascading failure: slowness spreads

Now the main event. The opening story continued like this:

Note that in step 1 Redis did not crash, it only slowed down. Had it crashed, the gateways would have received instant errors and responded quickly. Slowness is worse, because it makes everyone wait.

Three defences, in order

// The default is usually NONE or 30+ seconds.
// In trading, 30 seconds means infinity.
jedis.setTimeout(200);                        // Redis: 200 ms
httpClient.timeout(Duration.ofMillis(800));   // MT5 bridge
db.setQueryTimeout(3);                        // seconds

// If the queue is 80% full, return 503 immediately for non-critical requests
if (queue.utilisation() > 0.8 && !request.isCritical()) {
    return error(503, "Retry-After: 2");
}

4. Circuit breaker: stop calling what does not answer

The concrete incident: the MT5 bridge stopped responding. Every order held a thread until its 30-second timeout. The pool had 200 threads. In seven seconds the pool was exhausted and the gateway could no longer send orders even to healthy bridges.

So one bridge's failure stopped all order flow.

CircuitBreakerConfig.custom()
    .slidingWindowSize(100)              // last 100 calls
    .minimumNumberOfCalls(20)            // ← do not decide before 20 calls
    .failureRateThreshold(50)            // 50% failures -> open
    .slowCallDurationThreshold(Duration.ofMillis(800))
    .slowCallRateThreshold(50)           // ← SLOW calls count as failures
    .waitDurationInOpenState(Duration.ofSeconds(10))
    .permittedNumberOfCallsInHalfOpenState(5)
    .build();

5. Gray failure: alive but dead

The hardest failure class to diagnose. The streaming engine is up, /health returns 200 OK, the process is running. But because of a long garbage collection pause it cannot do the actual work: it is not processing prices, it is publishing the 4-second-old quote it already had.

The load balancer sees a healthy node and keeps sending traffic. Result: a quarter of orders open on stale prices.

GET /health
→ 200 {"status":"UP"}

// The code:
return ok();      // ← the process is alive, that is all

GET /ready
{
  "last_tick_age_ms": 4200,     // ← no tick processed for 4 seconds
  "queue_depth": 48000,
  "gc_pause_p99_ms": 1800,
  "status": "DEGRADED"
}

// Rule: if last_tick_age > 1000 then NOT READY

On top of that, outlier detection at the load balancer is very effective: a rule like "this node is 5× slower than the others" pulls it out temporarily even when its health check says OK. It is the most practical defence against gray failure, because it measures health relatively rather than absolutely.

6. Failover: it took over, but what did it lose?

The Redis master died, Sentinel promoted a replica, and the system came back in 8 seconds. Everyone relaxes. But:

The only real fix is architectural:

1. Critical state must not live only in a cache. The source of truth for orders and positions has to be a durable database, with Redis as a fast copy. The "let us keep it in Redis, it is fast" decision looks right until failover day.
2. Post-failover reconciliation must be automatic. On detecting a promotion, trigger a job that pulls the last N minutes of orders from the bridge, compares them with local state and reports the differences. Anything meant to be done manually does not get done.
3. Record the promotion as an event. "A failover happened at this time" is the only answer to "why is this order missing" three days later.

All together: which mode is which

Checklist AI agent task list

Conclusion

What these six modes have in common: none of them comes from a coding mistake. They all emerge from correctly written parts feeding each other under load. That is why they never show up in unit tests, only on a bad morning.

And the cure for all of them gathers in one place: set a limit. A limit on timeouts, on queues, on retries, on how much starts at the same moment. Anything without a limit will eventually have one set for it — usually by memory running out.