Contact me
All articles

Reliable Webhooks: Signatures, Retries, and Idempotency

Design a webhook receiver that verifies signatures, handles duplicate deliveries, and recovers from worker failures without repeating side effects.

Receiving an event is different from completing its work

A webhook lets another service notify your application through an HTTP request. A small demo may update a record immediately and return success. A production receiver also needs a policy for an interrupted connection, a repeated delivery, or a worker that fails after changing data.

Separate three milestones: the request has been authenticated, the event has been durably accepted, and its business action has completed. Recording those milestones makes a retry easier to understand. Without them, a successful HTTP response can hide unfinished work, while a timeout can trigger work that already happened.

Verify the signature before trusting the payload

Use the provider's documented signature verification method. Stripe requires the original request body for verification; parsing and reconstructing JSON can change the bytes and invalidate the check. Prefer the provider's SDK verification helper and configure the correct endpoint secret.

Signature verification does not replace application validation. After verification, check the event type and the fields needed for your action. Decide explicitly how unsupported events are handled. An accepted payload should never select an arbitrary operation merely because it contains a familiar-looking field.

Accept durably, then process asynchronously

GitHub recommends responding promptly and using a queue when processing needs more time. A practical receiver can verify the request, store an event or enqueue it durably, and acknowledge acceptance. A worker then performs the longer operation.

If the receiver cannot persist the event, returning success would risk losing it. If storage succeeds but queue publication fails, a recoverable handoff is needed. For example, a worker can poll pending events, or an outbox can publish them later. Choose one path that lets the system resume after a crash instead of relying on an in-memory promise.

Make duplicate deliveries safe

Stripe documents duplicate events and advises recording processed event IDs. It also does not guarantee delivery order. Treat delivery as something that can repeat, and avoid assuming the previous event has already arrived.

For a local database action, a unique event key and a transaction can keep the change and completion record consistent. External actions need extra care: if an email was sent before a worker crashed, recording the event only afterward can allow another email on retry. Use an external idempotency key when supported, or design a recoverable side-effect workflow. Deduplicating receipt alone does not make every action exactly-once.

Test the failure windows

Send the same valid event twice and verify that the intended state change occurs once. Send an invalid signature and confirm no event is accepted. Then interrupt processing after persistence but before completion and verify that pending work can resume. Include two different events for the same object to check that the deduplication key is not too broad.

Track pending, completed, and failed events with attempt counts and useful error details. Set bounded retries for transient failures and provide a review path for persistent ones. These observations help distinguish a delivery problem from a business-rule failure without treating every retry as a fresh event.

References

Stripe: webhook signatures, duplicates, and event order

GitHub: webhook best practices

Back to articles

Have something
you want to build?

Tell me about your idea, your project, or an opportunity to work together.