Home → Engineering
Distributed Transactions: Why 2PC Blocks and Saga Saves You
A two-leg order: buy EURUSD + sell GBPUSD. Both must happen or
neither. The first filled, the second was rejected.
And there is no ROLLBACK command available to you.
- The luxury of
ROLLBACKin one database does not exist in a distributed world. Trying to imitate it is the real mistake. - 2PC blocks. If the coordinator crashes everyone waits, locked. In trading that is unacceptable.
- A saga does not undo, it compensates. The difference is commercial: compensation has a cost.
- Simple flow, choreography; critical flow, orchestration. The test: can you answer "where is this order right now?"
In one database, this problem did not exist
When everything lived in one PostgreSQL, life was easy:
BEGIN;
UPDATE margin SET used = used + 1200 WHERE account = 42;
INSERT INTO orders (...) VALUES (...);
INSERT INTO orders (...) VALUES (...);
COMMIT; -- all or nothing
Now margin lives in one service, risk in another, and order routing goes through a
bridge to an external system. There is no single COMMIT,
and there will not be.
2PC: the textbook answer, the field problem
Two-phase commit was designed for exactly this. There is a coordinator and it asks two rounds of questions:
PHASE 1 — PREPARE
Coordinator -> margin-service : "are you ready?" -> READY (locked)
Coordinator -> risk-service : "are you ready?" -> READY (locked)
PHASE 2 — COMMIT
Coordinator -> everyone : "COMMIT"
...or if anyone said no : "ABORT"Flawless on paper. Now: what happens if the coordinator crashes between phase 1 and phase 2?
The margin service said "ready" and locked that account's margin. Now it is waiting for a decision. The coordinator is gone. What does it do?
- It cannot commit alone — maybe someone else said no.
- It cannot abort alone — maybe the others committed.
- So it waits. Until the coordinator returns.
Meanwhile that customer's account is locked. They cannot open a new order or close a position. The market is moving and the customer is staring at a screen. There is no scenario in a trading system where you can defend that sentence.
The second and more practical problem: external systems do not speak this protocol. You cannot tell an MT5 bridge or a liquidity provider "get ready, I will confirm shortly". They know one thing: send the order, take the result.
There is such a thing as 3PC: adding a pre-commit phase so that participants can proceed on a timeout when the coordinator dies. It reduces the blocking problem but can still make a wrong decision during a network partition and adds another round trip. I have never seen it used on the trading side; good to know about, unnecessary to build.
Saga: no undo, only compensation
The saga idea: split the long operation into small steps that complete on their own. Give each step a compensation. If something blows up, compensate backwards through what was already done.
The critical difference: steps commit immediately. Nobody holds a lock, nobody waits for anybody.
| Step | Forward | Compensation |
|---|---|---|
| 1 | Reserve margin | Release the reserved margin |
| 2 | Record the risk check | Cancel the record |
| 3 | Send leg 1 (buy EURUSD) | Open an opposite position — not undoable |
| 4 | Send leg 2 (sell GBPUSD) | Same |
| 5 | Notify the customer | Cannot be recalled — send a correction notice |
Look at the right-hand column: the first two are genuine undos. From the third onward there is no undo, only a business-level reversal. And it has a price.
Compensation: the most misunderstood part
"Compensating transaction" sounds like a clean ROLLBACK. It is not.
Concretely:
10:30:00.120 leg 1 filled: EURUSD BUY 1.0 lot @ 1.08450
10:30:00.480 leg 2 REJECTED: no liquidity
10:30:00.610 COMPENSATION: EURUSD SELL 1.0 lot @ 1.08442
Result: the position is closed but a 0.8 pip loss remains.
That loss did NOT disappear - somebody will absorb it.So compensation is not a technical operation but a decision with a commercial consequence. Who pays that difference is answered in a contract, not in code — and if that answer has not been written, there is no point writing the compensation code.
Three rules, all learned in the field:
- Put the step with no compensation last. Customer notifications, emails, accounting entries — these cannot be undone. Put them at the very end so no compensation is ever needed for them.
- Compensation can fail too. The market may have closed while you try to open the opposite position. There is no compensation for a compensation; at that point escalate to a human. Retrying forever makes it worse.
- Compensation must be idempotent too. If the compensation message arrives twice you open two opposite positions, and now you have actually moved the position the wrong way (details).
Choreography or orchestration?
Choreography: everyone knows their own part
No central coordinator. Each service does its work and publishes an event; the next service listens for it.
order-service : publish OrderCreated
margin-service : on OrderCreated -> publish MarginReserved
bridge-service : on MarginReserved -> publish OrderSent
notify-service : on OrderSent -> (send mail)Easy to set up, services do not know each other, adding a new listener is free. Genuinely good for simple flows.
A customer calls: "what happened to my order?" To answer you have to read the logs of six separate services, because the whole flow is written down nowhere. It is scattered across the listening rules of each service.
Worse: when a step fails, who runs the compensation chain? In choreography every service has to know, which undermines the independence it promised.
Orchestration: one coordinator, the flow in one place
A central orchestrator calls the steps in order and persists the state.
CREATE TABLE saga_state (
saga_id uuid PRIMARY KEY,
kind text, -- 'MULTI_LEG', 'OCO', 'BRACKET'
step text, -- 'MARGIN', 'LEG_1', 'LEG_2', 'NOTIFY'
status text, -- 'RUNNING', 'COMPENSATING', 'DONE', 'MANUAL'
done_so_far jsonb, -- everything needed to compensate
updated_at timestamptz
);
Thanks to this table, "where is the order?" is answered with a single
SELECT. And even if the orchestrator crashes, on restart it reads
unfinished sagas from the table and continues where it left off.
That is why state must be written to disk before every step. A state machine kept in memory forgets half-finished orders on the first restart and nobody picks them up.
Also, two orchestrator copies must not run the same saga — which is directly a leader election matter.
| Choreography | Orchestration | |
|---|---|---|
| Where the flow is written | Nowhere (scattered) | One place |
| "Where is the order?" | 6 service logs | One query |
| Compensation handling | Each service knows its own | Run centrally |
| Adding a step | Easy | The orchestrator changes |
| Single point of dependency | None | Yes (needs leader election) |
| When | Up to 3 steps, simple flow | Multi-leg, critical, money-touching |
For us the split settled like this: a normal single-leg order flows through choreography — three steps, everyone understands it. Multi-leg, OCO and bracket orders go through the orchestrator. The deciding question was: "If this flow stops halfway, can I read from somewhere what happened?"
Escrow: a resource held by a third party
A simple pattern that helps with things like transferring a position from one account to another. The resource does not go directly from A to B; an escrow account sits in between.
1. take from A -> put in escrow (A's balance dropped, B has not received)
2. give to B -> remove from escrow (transfer complete)
3. on failure -> return from escrow to AThe benefit: at no instant is the money in two places or in none. Where it is at any moment is written in a table. During reconciliation, the "sitting in escrow" rows are your list of half-finished transfers.
One condition: escrow entries must have a timeout. Otherwise a failed transfer sits there forever and nobody notices. A daily job should report anything that has been in escrow for more than an hour.
So, sagas everywhere?
No, and this matters. Sagas are complex; the setup, the compensation code and the monitoring all cost. Before using one, ask two questions:
- All steps in the same database → one transaction is enough
- Steps are independent → a queue is enough
- Stopping halfway is acceptable → retrying is enough
- Steps live in different services or systems
- Stopping halfway means money
- Every step has a meaningful compensation
A mistake I see often: building a saga inside a single service, with an orchestrator
written for three steps that all write to the same database. There
BEGIN ... COMMIT already exists and works perfectly.
A distributed transaction is only needed when you are actually distributed.
Checklist AI agent task list
- Is this really distributed, or would one transaction do?
- Does every step have a defined compensation?
- Are the steps with no compensation last?
- What happens if compensation fails? Does it reach a human?
- Is compensation idempotent?
- Is saga state written to disk, or kept in memory?
- Can "where is this order right now?" be answered with one query?
- With an orchestrator: can two copies run the same saga?
- Is there a report that finds half-finished sagas?
- Who absorbs the commercial cost of compensation? Is it written down?
Conclusion
The hard part of distributed transactions is conceptual, not technical: you have to give
up the ROLLBACK habit. In one database you can erase a past operation as
if it never happened; in a distributed world what happened has happened, and all you
can do is fix the state with a second operation.
So the best design is the one ordered to avoid compensation: put the riskiest, least-undoable step last and keep everything before it cheap and reversible. That was the fix for the two-leg order too — checking the second leg's liquidity before sending the first.