How Many Times Does a Message Arrive?

Why you have to choose a guarantee

Start with a simple situation: you sent a message and no reply came. What happened?

• The message never arrived.
• The message arrived, was processed, and the acknowledgement was lost on the way back.

On the sending side these two are indistinguishable. Both look the same: silence. And you have exactly two decisions available:

• Send again → the message is never lost but may be processed twice. That is at-least-once.
• Do not send → it is never processed twice but may be lost. That is at-most-once.

There is no third option. The whole subject comes out of that one ambiguity.

At-most-once: the lost tick problem

Price feeds usually arrive over UDP, and UDP promises nothing: if a packet drops it is gone and nobody resends it. The first time you hear this it sounds wrong — why not use TCP?

Because a repeated tick is useless. Getting the EURUSD price from 50 ms ago has no value; a newer one already arrived. The time TCP would spend retransmitting does not recover data, it delays the whole stream. Here loss is acceptable; latency is not.

class Quote {
    String  symbol;
    double  bid, ask;
    long    sourceTime;    // provider's stamp
    long    arrivalTime;   // when we received it
    long    sequence;      // increasing number from the provider
}

// BEFORE opening an order:
long age = now() - quote.arrivalTime;
if (age > STALE_THRESHOLD_MS) {      // e.g. 500 ms
    reject("stale quote: " + age + " ms");
}

At-least-once: the fill that arrived twice

Now the real subject. Orders, fills, balance movements — losing these is not acceptable, so you use at-least-once. Which means they will arrive twice.

How it happens, one by one:

All four are normal operation. None of them is a bug. So chasing "make sure it does not arrive twice" is the wrong path; the right one is answering "what happens if it arrives twice?"

Why exactly-once is a myth (and what actually exists)

When you hear "we support exactly-once", the question to ask is: in delivery, or in processing?

• Exactly-once delivery: not possible. The ambiguity above — was it the ack or the message that was lost — does not go away.
• Exactly-once processing: possible. And the method is known: at-least-once delivery + a consumer that discards duplicates.

Kafka's exactly-once falls into the second category, with one condition: both the read and the write must be inside Kafka. If you read from Kafka and write to PostgreSQL, that guarantee does not cover you — you are now in dual write territory.

Cure 1: an idempotency key (request path)

The side sending the order generates a unique key per logical order and sends the same key on retries.

-- Key table, unique constraint is essential
CREATE TABLE request_keys (
  key        text PRIMARY KEY,       -- client_order_id
  account_id bigint NOT NULL,
  result     jsonb,                  -- a copy of the first response
  created    timestamptz NOT NULL DEFAULT now()
);

BEGIN;
  INSERT INTO request_keys (key, account_id)
  VALUES (:client_order_id, :account)
  ON CONFLICT (key) DO NOTHING
  RETURNING key;
  -- If no row came back: this request was already processed.
  --   -> DO NOT act, return the stored result.

  -- If a row came back: first time, place the order
  INSERT INTO orders (...) VALUES (...);
  UPDATE request_keys SET result = :response WHERE key = :client_order_id;
COMMIT;

Two subtleties, both learned the hard way:

• The client generates the key, not you. Generate it server-side and every retry produces a new key, so the protection never fires.
• Store the first response and return it again. If you return an "already exists" error, the client reads it as a failure and retries again. The second request must receive the first request's answer.

Cure 2: an idempotent consumer (event path)

For a fill notification arriving on a queue there is no client; the key travels inside the event (fill_id, deal_id, event_id). The consumer checks whether it has seen that id.

BEGIN;
  INSERT INTO processed_events (event_id, kind)
  VALUES (:fill_id, 'FILL')
  ON CONFLICT (event_id) DO NOTHING
  RETURNING event_id;
  -- No row? This fill was already processed -> COMMIT and exit

  -- First time: do the real work
  INSERT INTO fill_records (...) VALUES (...);
  UPDATE positions
     SET lots = lots + :lots, avg_price = ...
   WHERE id = :position_id;
COMMIT;

The dedup table does not grow forever

With millions of events a day, that table becomes your main load a year later. Practical approach:

• Pick a window. Duplicates arrive within minutes, not months. Seven days is more than enough.
• Clean up the old. A daily job: DELETE FROM processed_events WHERE created < now() - interval '7 days'. Index the date column, otherwise the cleanup locks the table.
• Think about outside the window. A duplicate arriving after the window will be reprocessed. If that is unacceptable, add a unique constraint on the real table too — a second line of defence.

So which guarantee for which data?

The easiest way to decide is one question: is losing this message worse, or processing it twice?

Note: both exist in the same system, and they should. Making the price feed at-least-once kills latency; making orders at-most-once kills money. Standardising on one guarantee means being wrong on one of the two sides.

From the field: how we closed the double fill

Back to the opening incident. The same fill arrived twice and two records were created. The first instinct was "fix the bridge so it stops sending twice". Wrong instinct: the bridge was behaving correctly, resending because it got no ack.

What we actually did:

1. A unique constraint on deal_id. Not a check in the application, a prohibition in the database. However many copies run, the rule lives in one place.
2. Dedup and the position update moved into the same transaction. They had been separate, and a crash in between had once produced "processed but not written" — harder to spot than a duplicate.
3. The duplicate counter became a metric. duplicate_events_total is not zero and we do not expect it to be. But when the number jumps, it is usually the first sign of a problem on the bridge or the network. So dedup is not only a guard, it is also a sensor.

Checklist AI agent task list

Conclusion

Choosing a delivery guarantee looks like a technical preference but it is really a commercial one: what are you willing to lose? You accept loss on prices because a new one is coming. You do not accept it on fills because that is money.

And one sentence to keep: exactly-once is not a delivery guarantee, it is a consumer design. Nobody can sell it to you; you write it yourself, in about eight lines of ON CONFLICT DO NOTHING.