Race Conditions: Why Does Stock Go Negative?

What is a race condition?

The definition is dull: two operations touch the same data at once and the result depends on which finishes first. But in the field it always shows up as the same pattern:

product = SELECT * FROM products WHERE id = 42;

if (product.stock > 0) {           // ← check
    // ... milliseconds pass here ...
    UPDATE products SET stock = product.stock - 1 WHERE id = 42;   // ← act
    createOrder();
}

This code is flawless with one user. With two:

Note that B wrote using the stale value A had read. This is called a lost update, and stock does not even always go negative — sometimes there is simply one extra order, which is much harder to spot.

The same bug in four disguises

1. A single-use coupon used twice

The code checks "has this coupon been used?", applies the discount if not, then marks it used. If the user impatiently double-clicks, two requests go out almost simultaneously. Result: one coupon, two discounts.

2. A balance split in two

The wallet has 100. Two parallel withdrawal requests arrive, each for 80. Both pass the "sufficient balance" check. Result: a balance of −60. This is the class of bug that makes people sweat in a corporate setting.

3. A payment webhook delivered twice

A payment provider retries when it does not get a timely response — that is by design, not a bug. If you check "does this payment exist?" and insert if not, two simultaneous deliveries create two records. The customer gets two items or is charged twice.

4. Two workers picking up the same job

If two processes pulling from a queue run the "pending jobs" query at the same moment, both can pick the same row. The same email goes out twice. There is more on this in worker sharding.

Why did you never see it in staging?

Because staging has one user, one request, one instance. The bug requires two requests to collide within milliseconds. On top of that:

• Production runs multiple copies. An in-process lock works on one machine and means nothing on the second instance.
• Traffic is not flat, it clusters. Sale start, batch notifications, hourly jobs — all pile into the same second.
• Slowness widens the window. As the system slows under load, the gap between "check" and "act" grows and the bug becomes more likely. That is, it happens most at the worst possible moment.

Fixes, from cheapest to most expensive

1. Let the database decide (try this first)

The cheapest and most robust fix, and surprisingly under-used. Take the condition out of the if and put it in the query:

UPDATE products
   SET stock = stock - 1
 WHERE id = 42
   AND stock > 0;

-- if 1 row affected: stock decremented, create the order
-- if 0 rows:         out of stock, tell the user

Two things changed at once. First, we wrote stock = stock - 1, which uses the database's current value rather than the stale one from the application. Second, the condition lives in the same statement, so there is no gap between check and update.

Critical detail: you must check the return value. If zero rows were affected, nothing happened and you must not create the order. Skipping that line is the most common way to apply the fix and still keep the bug.

2. A unique constraint

The cleanest fix for the coupon case. Instead of asking "has it been used?", try to write the usage into its own table:

CREATE UNIQUE INDEX ux_coupon_usage ON coupon_usage (coupon_id);

-- in the application:
INSERT INTO coupon_usage (coupon_id, order_id) VALUES (7, 991);
-- success        -> apply the discount
-- unique violation -> "this coupon has already been used"

The idea here is elegant: instead of trying to prevent the bad state, you make it impossible in the database. However many copies of the application run, the rule lives in one place and is enforced.

3. A version column: optimistic locking

Useful for longer edits — two people opening the same record and saving. Add a version column:

UPDATE orders
   SET status = 'approved', version = version + 1
 WHERE id = 991
   AND version = 3;     -- the version I saw when reading

-- 0 rows affected: someone changed it before me
-- tell the user "the record changed, please review"

It is called optimistic because it assumes conflicts will not happen and detects them when they do. Where conflicts are rare it is the fastest approach; where they are common it keeps sending users to a "please try again" screen.

4. A row lock: pessimistic locking

For short but indivisible operations that touch several tables:

BEGIN;
SELECT balance FROM wallets WHERE id = 5 FOR UPDATE;  -- lock the row
-- no other transaction can read this row here
UPDATE wallets SET balance = balance - 80 WHERE id = 5;
INSERT INTO transactions (...) VALUES (...);
COMMIT;                                                -- lock released

5. An idempotency key

The right answer to repeated requests. The client (or the payment provider) generates a stable key per logical operation and you store it in a table with a unique constraint. When the same key arrives again, you do not repeat the work; you return the first result.

This removes the duplicate payment record entirely and also settles the "did it go through?" question during network failures.

6. A distributed lock

Last resort. Only if none of the above works — for example when the thing you are protecting is not a database row, or the operation spans several systems. It has its own traps and deserves its own post.

How do you notice it in production?

Race conditions do not produce error messages; they produce impossible data. So instead of searching for them, write queries that report impossible states:

-- fields that must never be negative
SELECT * FROM products WHERE stock < 0;
SELECT * FROM wallets  WHERE balance < 0;

-- records that must be unique
SELECT payment_ref, COUNT(*) FROM payments
GROUP BY payment_ref HAVING COUNT(*) > 1;

-- two records from the same user in the same second
SELECT user_id, COUNT(*) FROM orders
WHERE created > now() - interval '7 days'
GROUP BY user_id, date_trunc('second', created)
HAVING COUNT(*) > 1;

Pinning these three somewhere and checking weekly is the cheapest way to catch a bug that has been quietly corrupting data for months.

Reproduce it before you fix it

See the bug at least once. The simplest way is parallel requests:

seq 20 | xargs -P 20 -I{} \
  curl -s -X POST https://api.local/orders \
       -d '{"productId":42,"qty":1}' -o /dev/null

# then: SELECT stock FROM products WHERE id = 42;
# stock was 1 and you sent 20 requests. How many orders exist?

Run it once before the fix and once after. If you see a single order the second time, you are done. Skipping this step and saying "I think it is fixed" is the classic way to meet the same bug again in three months.

Checklist

Conclusion

Race conditions are not clever bugs; they come from the same pattern every time. Once you learn to recognise that pattern in code, you catch most of them before they are written.

One sentence to keep: if you check a condition in the application and enforce it in the database, there is a gap in between. The cheapest way to close it is to move the condition into the query.