Leader Election: Who Does This Job?

Where the problem comes from

Scaling a service horizontally usually feels free: two copies do twice the work. But not all work divides. Some jobs must happen exactly once, and they break quietly as the copy count grows.

Examples that will feel familiar from the trading side:

The right-hand column looks scarier at first, but the left one is more insidious: it produces no error, it just costs money. Health checks going out from four copies is invisible on day one; it becomes visible when the bridge provider starts rate limiting you.

Three easy but wrong answers

1. "Let's just run one copy"

It works — until that copy dies. Kubernetes starts a new one, 20–60 seconds pass, and during that window nobody health-checks anything. Acceptable? For some jobs, yes. For bridge health checks, no.

2. "Let's flag it with an environment variable"

if (os.getenv("IS_LEADER") == "true") {
    startHealthCheckLoop();
}

3. "First come first served, put a flag in Redis"

Right direction, half the answer. If you set a flag with SETNX leader 1 and no expiry, the flag survives a crash forever and nobody can become leader. The full story is in the distributed lock post; the difference here is that leadership is not a lock but a timed lease.

The right model: leadership is leased

The lease idea: leadership is not granted forever, it is rented for a period. The leader renews the rent regularly. If it cannot renew — because it crashed, got partitioned or paused — the lease expires and someone else takes over.

CREATE TABLE leadership (
  job_name   text PRIMARY KEY,
  owner      text NOT NULL,
  expires_at timestamptz NOT NULL,
  epoch      bigint NOT NULL DEFAULT 0    -- ← fencing token
);

-- Every copy runs this every 5 seconds:
INSERT INTO leadership (job_name, owner, expires_at, epoch)
VALUES ('bridge_health', :me, now() + interval '15 seconds', 1)
ON CONFLICT (job_name) DO UPDATE
   SET owner      = :me,
       expires_at = now() + interval '15 seconds',
       -- epoch increments ONLY when the leader changes
       epoch      = leadership.epoch + CASE WHEN leadership.owner = :me THEN 0 ELSE 1 END
 WHERE leadership.owner = :me            -- either it is already mine (renewal)
    OR leadership.expires_at < now()     -- or it expired (takeover)
RETURNING owner, epoch;

-- If the returned owner is me: I am the leader, carry on.
-- If no row came back: someone else leads, wait.

Thirty lines and no extra infrastructure. Three details matter:

• The lease must be at least 3× the renewal interval. Above: renew every 5 seconds, lease 15 seconds. Miss two renewals and I am still leader; on the third I lose it. A tighter ratio means a single network hiccup hands leadership over for nothing and the system flaps.
• Takeover cannot happen before expiry. The two conditions in the WHERE clause guarantee that.
• The epoch column is not decoration. That is the next section.

Split-brain: two leaders at once

Now the bad news. The code above is correct but it does not prevent two copies from believing they are the leader. And no code can. Here is why:

A did nothing wrong; it just slept and does not know the world moved on. This is split-brain and in distributed systems it is unavoidable.

The fix: an epoch number (fencing token)

Produce an increasing number on every leadership handover and make the protected resource store it. The resource then rejects anything carrying a lower number:

-- A (epoch 7) wakes up and tries to update bridge status:
UPDATE bridge_status
   SET health = 'DOWN', checked_at = now(), epoch = 7
 WHERE bridge_id = 'LP-3'
   AND epoch <= 7;          -- ← the table says 8, condition fails

-- 0 rows affected. A could not write.
-- The application sees that, concludes "I am no longer leader", stops the loop.

The subtlety: A does not have to ask whether it is still the leader. It tries to write, gets rejected, and learns. Correctness now rests on order rather than on timing — the same idea as in the distributed lock post.

Not needed for side-effect-free work

Not every job needs fencing. The dividing question: does the job leave a mark outside?

Do not write this yourself (usually)

The Postgres approach above is instructive, but in reality you probably already have a tool for this:

Writing your own Raft is almost never the right answer. That code looks correct, passes tests, and abandons you six months later during a network partition.

From the field: what we did with the health check

Back to the opening problem: four streaming engines, all polling the bridges. Three steps:

1. Leadership moved to a lease. 5-second renewal, 15-second lease. Ping traffic dropped to a quarter.
2. An epoch number was added to status writes. A copy coming out of a pause and trying to write "LP-3 is down" is rejected. Before that change, a copy waking from a pause marked a healthy bridge as dead and cut traffic to it — the most expensive lesson of the lot.
3. Leadership became a metric. Every copy publishes "am I the leader". The sum on the graph must be constantly 1. If it drops to 0, nobody is doing the job; if it hits 2, there is a split-brain. One graph, both failures.

DELETE FROM leadership WHERE job_name = 'bridge_health' AND owner = :me;

How many leaders? One — but for what?

A common design mistake: electing one global leader and loading every exclusive job onto it. The result is a single bottleneck inside a system that otherwise scales horizontally.

Better: elect leadership per job. One leader for health checks, another for reconciliation, another for symbol sync. The load spreads, and one job's leader getting stuck does not affect the others.

Take that one step further and you are in sharding territory: leadership says "let one copy do it", sharding says "let every copy do its own share". They are not rivals; you pick based on whether the work divides.

Checklist AI agent task list

Conclusion

Leader election sounds academic, but the practical version is a very simple question: who is going to do this job? Answer "all of them" and you get noise; answer "none of them" and you get a silent failure.

One thing to keep: leadership is not property, it is a rental. It expires, it changes hands, and the previous holder may not know. The robustness of your system is measured by what that previous holder can still do when it wakes up late.