Every digital-goods storefront has one problem that matters more than any other: a purchased key has to be reserved exactly once. Sell the same key twice and you have a furious customer and a chargeback. Take someone's money and fail to assign a key at all and you have a worse one. This is the core engineering problem behind Increddy's order fulfillment, and it's a concrete distributed-systems story worth walking through in detail rather than the usual "we built a checkout" summary.
Why this isn't a single-step problem
The naive design reserves a key the moment an order is placed. That breaks
the first time a payment webhook retries, which it will, because retries
are how payment providers handle their own transient failures. A retry
means order.placed or payment.captured can each fire more than once for
the same order, and the two events don't always arrive in the order you'd
expect. Any reservation logic written as "do this once, synchronously, at
one point in the flow" will eventually double-reserve or silently drop a
reservation the first time reality doesn't cooperate.
Two independent triggers, one idempotent write
Increddy reserves a key on order.placed as the primary path. A second,
fully independent trigger on payment.captured acts as a safety net in
case the first is missed or arrives out of sequence. Because both triggers
can legitimately fire for the same order, and because either one can retry,
the reservation write itself has to be idempotent at the data layer: a
second attempt to reserve for an order that already has a key assigned is
a no-op, not a second key handed out.
This is the part that's easy to get wrong in a code review: idempotency has to live in the write itself, not in application logic that checks "has this already happened" before writing, because that check-then-write pattern has its own race condition between the check and the write.
The reconciliation job: a second line of defense
Even with dual triggers, there's a small window of cases that fall through both: a payment captured with no order event ever firing being the clearest example. A scheduled job runs every 15 minutes, sweeps for any payment captured more than 5 minutes earlier with no key assignment yet, and closes that gap. It's not a workaround for a design flaw. It's a deliberate acknowledgment that any event-driven system has a tail of cases that need a sweep, not a trigger, and building that sweep in from day one beats discovering the need for it from a support ticket.
Building something with a similar race-condition risk?
If your product has a 'exactly once, no exceptions' moment like this, we'd rather talk through the failure modes with you before the build than after.
Talk to an engineer →Overselling, allowed on purpose
The least intuitive decision in the whole system: a sold-out product can still be purchased. Blocking the sale outright on an inventory count risks rejecting a legitimate buyer over a count that's already stale by checkout. Instead, an oversold order drops into a manual-assignment queue for an admin, who can source or assign a key by hand. For a digital-goods business, a short manual-resolution delay is a far better failure mode than a false "sold out" turning away a paying customer.
The rule
Model the state transition (key assigned or not) as the thing you make idempotent, not the business event that triggers it. Build a reconciliation sweep as a first-class part of the design, not a patch added after the first incident. And when a hard failure mode is unavoidable, choose the one that costs you an admin's time over the one that costs you a customer. For the migration this sat inside, see how we replaced Shopify's backend without touching the frontend. We build this kind of order-fulfillment logic as part of Medusa development engagements.