Webhooks that don’t lose money: idempotency, retries and reconciliation

A webhook is a promise from another system: "something happened, here are the details." Networks, queues and retries mean the promise can be delivered more than once, out of order, or not at all. Systems that move money have to be designed for all three.

Idempotency: process each event once

Every incoming event should carry an identifier — a transaction reference, an event id. Store it before acting, and ignore any repeat. If the provider does not give you one, derive it from stable fields. The goal is that a duplicate delivery changes nothing.

Retries: on both sides

Providers retry when your endpoint fails, so respond quickly and do the work in a queue. Your own outbound calls — status queries, partner notifications — need retries with backoff too, and a dead-letter list for anything that keeps failing so a human sees it.

Reconciliation: trust, then verify

Even a perfect webhook handler drifts from the provider's records over time. A scheduled job that compares your ledger with the provider's statement — by reference, amount and date — catches the missing callback, the reversed transaction and the fee you forgot to book. Unmatched items become a worklist, not a month-end surprise.

Make state explicit

"Initiated, pending, confirmed, settled, failed" — a small state machine with allowed transitions prevents the classic bug of a confirmation arriving for a transaction already marked failed. Every transition is logged, and the current state is always explainable.