Worker Sharding: The Right Way to Split Work

First: is a queue not enough?

In most cases it is, and sharding gets written for nothing. The distinction is simple:

The key sentence: the point of sharding is not speed, it is removing collisions. With locking, everyone runs to the same door and queues; with sharding, they were already entering through different doors.

Method 1: modulo (easiest, most fragile)

shard = hash(user_id) % WORKER_COUNT

# worker 2 only processes:
SELECT * FROM jobs
 WHERE status = 'pending'
   AND MOD(hashtext(user_id::text), 4) = 2
 ORDER BY created
 LIMIT 100;

It works, there is no lock, the code is short. Trouble starts when you change scale.

Method 2: a fixed number of logical shards (best in practice)

My favourite solution and surprisingly simple: decouple the shard count from the worker count. Pick a fixed, generous number — say 64 — and hand out shard numbers to workers.

SHARD_COUNT = 64          # never changes
shard = hash(user_id) % 64

# with 4 workers:
worker 0 -> shards 0..15
worker 1 -> shards 16..31
worker 2 -> shards 32..47
worker 3 -> shards 48..63

# add a 5th worker and the shards are redistributed,
# but a key's SHARD does not change. Only its owner does.

The difference is subtle and important: the key → shard mapping stays fixed and only the shard → worker mapping changes. That makes per-shard state, caches and progress information transferable. Kafka's partition model is the same idea.

One rule when choosing the shard count: make it several times the number of workers you can imagine ever running. 64 or 128 is plenty for most systems.

Consistent hashing: how to hand shards to workers

The fixed-shard approach leaves one question open: how do you distribute 64 shards across 4 workers? The easiest way is ranges (0–15, 16–31...), but when a fifth worker appears those ranges are recomputed and nearly every shard changes hands.

Consistent hashing solves exactly that. The idea: place both the workers and the keys on an imaginary ring. Each shard belongs to the first worker clockwise from it.

Ring (0 .. 2^32):

   worker-A ●          ● worker-B
       ↘  shards 7,12,31  ↙
            ●
        worker-C

# When worker-D is added:
# Only the shards of D's NEIGHBOUR on the ring move.
# Everything else stays untouched.

owner(shard) = the first worker clockwise from hash(shard)

A practical detail: each worker must be placed on the ring at 100–200 virtual points, not one. With a single point the distribution is left to luck and one worker can end up with three times another's load. Virtual nodes flatten it, and most libraries do this by default.

Why it matters: the reconnect storm

Rebalancing: hand over gradually

Consistent hashing reduces how many shards move but does not make it zero. The ones that do move should be handed over one at a time, not all at once:

for (shard : shardsToMove) {
    oldOwner.stop(shard);           // stop taking new work, finish what you hold
    oldOwner.confirmDrained(shard); // ← wait for this
    newOwner.start(shard);
    Thread.sleep(200);              // ← breathe before the next shard
}

Method 3: a claim table (dynamic and visible)

Instead of fixed shards, workers "claim" the work themselves, in one atomic update:

UPDATE jobs
   SET owner = :worker_id,
       claim_expires = now() + interval '5 minutes',
       status = 'processing'
 WHERE id IN (
       SELECT id FROM jobs
        WHERE status = 'pending'
           OR (status = 'processing' AND claim_expires < now())  -- a dead worker's job
        ORDER BY created
        LIMIT 50
        FOR UPDATE SKIP LOCKED     -- ← the key line
 )
RETURNING *;

SKIP LOCKED is the trick: it skips without waiting any rows another worker has locked. So workers pick different jobs without ever blocking each other.

The second benefit: thanks to claim_expires, a dead worker's jobs free themselves after five minutes. You do not need a separate health check.

The cost is the loss of ordering: two jobs for the same user can land on different workers. If order matters, this method is not for you.

Two cases from the field

1. One customer, four workers, a hash that did nothing

A notification system was split by customer_id % 4. One enterprise customer alone produced about 40% of the daily notifications.

Result: the worker that customer hashed to was permanently at 100% CPU while the other three sat idle. We raised the worker count to 8 — nothing changed, because that customer still landed on exactly one worker.

-- before
shard = hash(customer_id) % 64

-- after
shard = hash(customer_id + ":" + notification_id) % 64

2. The day ordering broke

Order status updates were written to a queue: preparing → shipped → delivered. As load grew, the worker count was increased, and a week later support reported something odd: some orders were going back from "delivered" to "shipped".

The reason was straightforward: two messages for the same order had landed on two different workers, and the "shipped" message was processed after the "delivered" message that was produced later.

We fixed it in two layers:

1. Shard by order id. All messages for the same order go to the same worker and are therefore processed in order.
2. Guard the transition. Even with sharding, a rebalance can break the order one day, so let the database protect itself too: UPDATE ... WHERE status_rank < :new_rank. A late old message cannot write.

The second point is the same idea as a fencing token: entrust correctness to order, not to timing.

Rebalancing: the most dangerous moment

When you add or remove workers, shards change hands. In that exact second two workers can believe they own the same shard. It is sharding's most critical moment and its least tested part.

Three things worth measuring

That third row matters: do not look at the average. With three of four workers idle, average latency looks great; the customer on the congested shard gets their notifications half an hour later.

Checklist AI agent task list

Conclusion

Sharding is a more elegant answer than locking: instead of managing collisions, it removes them. But it is not free — you have to think about scale changes, dead workers and uneven data.

Back to the opening picture: the reason those three workers were idle was not capacity, it was the choice of key. The hard part of sharding is not writing the code but choosing the right key — and the right key is always chosen by looking at the real distribution of the data, never by guessing.