Queue Pathologies

1. The poison message: one line that locks the queue

An order message arrived and cannot be parsed. Maybe a field is missing, maybe volume holds an empty string instead of a number, maybe the encoding is broken. The consumer throws, does not ack, the message stays in the queue and is redelivered.

Then the same thing happens. And again. And again. This is a loop spinning hundreds of times a second, with 40,000 orders queued behind it.

The cure: a dead letter queue

The rule is simple: if a message has been tried N times, stop trying and move it aside.

void handle(Message m) {
    try {
        processOrder(m);
        ack(m);
    } catch (PermanentError e) {      // parsing, validation, business rule
        toDlq(m, e);                  // ← no retry at all, straight aside
        ack(m);
    } catch (TransientError e) {      // network, timeout, 503
        if (m.attempts >= 5) {
            toDlq(m, e);
            ack(m);
        } else {
            m.attempts++;
            toDelayQueue(m, backoff(m.attempts));
            ack(m);                   // REMOVE from the main queue
        }
    }
}

Two distinctions matter, and most teams skip the first:

• A permanent error is never retried. A message that cannot be parsed will still not parse on the fifth attempt. Retrying just burns time and resources. Got a 400 from a server? Do not retry. A 503? Do.
• A message to be retried is removed from the main queue. It moves to a separate delay queue. Leave it in place and "retrying" becomes the same thing as blocking everyone behind it.

2. Exponential backoff — and why jitter matters

Retrying immediately on a transient error makes things worse: if the other side is already struggling, a second request strains it further. The classic answer is to double the wait each time:

100 ms → 200 ms → 400 ms → 800 ms → DLQ

Correct, but incomplete. Consider: the bridge went down for a second and 800 orders failed at the same moment. All wait 100 ms and retry at the same moment. All fail again, all wait 200 ms and retry at the same moment again.

So instead of random load you have produced synchronised waves. The bridge, trying to recover, gets slapped by 800 requests at regular intervals.

long backoff(int attempt) {
    long base = Math.min(BASE_MS * (1L << attempt), CAP_MS);   // 100,200,400,800...
    // "full jitter": random between 0 and base
    return ThreadLocalRandom.current().nextLong(base + 1);
}

3. Head-of-line blocking: big orders delay small ones

There is a single Redis stream and every order flows through it. An enterprise customer sends a 1000-lot order; it is split into 40 pieces across bridges and takes 2 seconds to process.

For those 2 seconds, 0.01-lot orders behind it wait. Average latency on the dashboard looks fine while the 99th percentile goes through the roof.

What we did was a mix of the first and the fourth: orders above 100 lots moved to a separate lane and were chunked there. Small-order latency dropped from 2100 ms to 40 ms at the 99th percentile — the average had always been fine; the problem lived entirely in the tail.

4. Reordering: #3 first, #1 later

Partial fills are sent in order: #1 (10 lots), #2 (15 lots), #3 (5 lots). They arrive as #3, #1, #2. The cumulative calculation is momentarily wrong and the customer sees strange numbers on screen.

Why does it happen?

• The messages landed on different partitions and were handled by different consumers.
• #1 failed once and was retried, while #2 and #3 went past it.
• Different paths and different delays on the network.

Three fixes, weakest to strongest

// Expected: 1. Arrived: 3.
if (m.seq > expectedSeq) {
    pending.put(m.seq, m);               // hold it aside
    if (pending.oldestAge() > 200) {     // waited more than 200 ms
        warn("missing message: " + expectedSeq);
        expectedSeq = pending.firstKey(); // skip, do not deadlock
    }
    return;
}
process(m);
expectedSeq++;
drainPending();

// Fragile — order required, sends a delta
{ "order_id": 991, "fill_no": 2, "added_lots": 15 }

// Robust — order irrelevant, sends absolute values
{ "order_id": 991, "fill_no": 2,
  "cumulative_lots": 25, "cumulative_amount": 27512.50,
  "remaining_lots": 5 }

UPDATE positions
   SET cumulative_lots = :cumulative_lots,
       last_fill_no    = :fill_no
 WHERE order_id = 991
   AND last_fill_no < :fill_no;     -- ← an old message cannot write

The difference is at the design level: a system sending deltas depends on every message arriving, in order; a system sending absolute values recovers with a single message. When designing a message schema, the question to ask is: "If I received only this message, could I reach the correct state?"

5. Backpressure: 50k ticks and a consumer that cannot keep up

The market got busy. 50,000 ticks a second are arriving and the streaming engine can handle 30,000. Where do the other 20,000 go? You have three options, and the third is not an option:

The third is the most common, because it is the most invisible when writing code. Creating an unbounded queue is one line, and that line makes it uncertain when your system will die. Put an upper bound on every queue and decide consciously what happens when it fills.

For prices: not dropping, but conflating

There is a nice trick on the price side. If 40 ticks for EURUSD have piled up, there is no point processing all 40 — the customer will only see the last one. So instead of a queue, keep the latest value per symbol:

// Not a queue — one slot per symbol
ConcurrentHashMap<String, Quote> latest = new ConcurrentHashMap<>();

// Producer: overwrite, do not accumulate
latest.put(q.symbol, q);

// Consumer: read at whatever speed it can manage
for (var e : latest.entrySet()) {
    publish(e.getValue());
}

For orders: a bounded queue and an honest rejection

Orders cannot be dropped. But they cannot be buffered without limit either. The right behaviour when the queue fills is to reject the new order openly:

BlockingQueue<Order> queue = new ArrayBlockingQueue<>(10_000);

if (!queue.offer(order)) {                 // full
    metrics.increment("orders_rejected_queue_full");
    return error("SYSTEM_BUSY, please retry");
}

What to watch

Checklist AI agent task list

Conclusion

A queue is a cushion between systems. But a cushion is also a concealer: it hides the slowness behind it for a while, then hands you all of it at once.

That 40,000-message queue came from a single malformed order. The fix was fifteen lines. The real lesson was that those fifteen lines should have been written on day one — because a poison message is not a possibility, it is a certainty.