Distributed Locks: Getting It Right

Stop: do you really need this?

The distributed lock is a good example of everything looking like a nail once you hold a hammer. In most cases there is a cheaper answer:

The rule: if the thing you are protecting is a single database row, you do not need a lock. Locks become meaningful when what you must protect lives outside the database.

Step by step: from wrong to right

Attempt 1 — the innocent version (broken)

if (redis.GET("lock:invoice") == null) {
    redis.SET("lock:invoice", "1");   // ← there is a gap here
    doWork();
    redis.DEL("lock:invoice");
}

Attempt 2 — atomic acquire, no expiry (broken)

SETNX lock:invoice 1      # atomic, good
# ... work happens ...
DEL lock:invoice

Attempt 3 — expiry, but in two commands (still broken)

SETNX lock:invoice 1
EXPIRE lock:invoice 60     # ← second command

Attempt 4 — correct acquisition

token = uuid()                                  # unique to this copy
SET lock:invoice <token> NX PX 30000

# NX  -> write only if absent (atomic)
# PX  -> delete itself after 30 seconds
# token -> so we can say "this lock is mine"

That token is a lifesaver, and the subject of the next section.

The real trap: releasing the lock

Most teams acquire correctly. Almost none release correctly. Look at this sequence:

A deleted B's lock. Now B and C run at the same time and nobody knows.

-- Redis Lua: runs atomically
if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("DEL", KEYS[1])
else
    return 0
end

What if it expires?

The real problem in that table is not the deletion: B started while A was still running. Compare-and-delete does not solve that, it only reduces the damage.

Partial fix: renew the lock

-- every 10 seconds, extend only if it is still mine
if redis.call("GET", KEYS[1]) == ARGV[1] then
    return redis.call("PEXPIRE", KEYS[1], ARGV[2])
else
    return 0
end

The real fix: fencing tokens

This is the most important and least known part of the topic. The idea: every time the lock is granted, also hand out an increasing number. The protected resource then rejects any write carrying a lower number than it has seen.

-- on the Redis side: a counter that increments on each acquisition
INCR lock:invoice:counter        -- returns 41 for A, 42 for B

-- send the number along when writing:
UPDATE invoice_state
   SET last_run = 'done', fence = 42
 WHERE id = 7
   AND fence < 42;               -- ← rejects the old owner

-- if A (fence=41) wakes up late and tries to write:
-- the condition fence < 41 does not hold, 0 rows affected, it cannot write.

The beauty is this: whether A wakes up, how late it is, whether the lock expired — none of it matters any more. The protected resource itself rejects the stale owner. Lock safety no longer depends on timing.

There is a cost: the protected resource must be able to perform that check. Easy in a database, usually impossible on a file system or a third-party API. If it is impossible, at least you know you are living with it.

About Redlock

There is an algorithm that proposes acquiring a lock across several Redis nodes at once, and a long debate about it. In short: a single Redis node is a single point of failure, but multiple nodes do not give a full guarantee either, because of clock drift and pauses.

My practical position:

• If you lock for efficiency (let us not do the same work twice, let us not waste resources), a single Redis node is fine. A rare overlap only means waste.
• If you lock for correctness (if this happens twice we lose money), do not trust the lock alone. Add a fencing token or make the work idempotent.

Saying "we added a lock, we are safe" without making that distinction is the most expensive misconception in this area.

Three simpler options than Redis

1. Postgres advisory locks (free if you already use it)

-- held for the session, released AUTOMATICALLY if the connection drops
SELECT pg_try_advisory_lock(834721);
-- true  -> the lock is mine, start work
-- false -> someone else is running, exit

SELECT pg_advisory_unlock(834721);   -- when done

2. A lock table (works on any database)

CREATE TABLE locks (
  name       text PRIMARY KEY,
  owner      text NOT NULL,
  expires_at timestamptz NOT NULL
);

-- acquire: take over if expired
INSERT INTO locks (name, owner, expires_at)
VALUES ('invoice', :me, now() + interval '30 seconds')
ON CONFLICT (name) DO UPDATE
   SET owner = :me, expires_at = now() + interval '30 seconds'
 WHERE locks.expires_at < now()
RETURNING owner;

-- if the returned owner is me, the lock is mine

3. No lock at all: a single-consumer queue

Most "only one at a time" needs are really ordering needs. Put the work in a queue and have a single consumer process it; no lock required. If you run on Kubernetes, lease-based leader election does the same job for scheduled work without you writing it.

From the field: what we did with the invoicing job

Back to the opening story. The first instinct was to add a Redis lock. We did, and the problem disappeared for a while. Then one day the job ran longer than expected, the lock expired, and the same bug came back — less frequently this time, which made it harder to notice.

The permanent fix had two parts, and neither was about locking:

1. We made invoicing idempotent. A unique constraint on (period, customer_id). A second attempt is rejected by the database; duplicate invoices are impossible with or without a lock.
2. We kept the lock for efficiency. We moved to an advisory lock; its purpose is not correctness but stopping two copies from doing the same work for nothing.

The lesson stayed with me: a lock does not make a broken design safe. The work itself has to be repeatable; the lock only reduces waste.

Checklist AI agent task list

Conclusion

The distributed lock is one of the easiest things to get wrong in distributed systems, precisely because the wrong version usually works. The bug only appears when load, latency and luck line up — that is, at the worst moment.

The order to remember: first design so you do not need a lock; if you cannot, use the database's lock; if you cannot do that either, write a Redis lock — but entrust correctness to a fencing token or idempotency, not to the lock.