If you run outbound webhooks in production, you must design for failure. Not because you want complexity. Because networks fail and receivers go down.
Start here:
The anti-pattern: "fire and forget"
When outbound webhooks are "fire and forget":
- deliveries fail silently
- external systems drift out of sync
- support teams cannot prove what happened
The result is usually manual reconciliation work and distrust in the integration.
A bounded retry strategy (minimum viable)
You want retries, but you want them bounded:
- Max attempts (initial + retries)
- Backoff schedule (e.g., seconds/minutes, not constant rapid retry)
- A final terminal state when you give up
The terminal state is commonly called "dead letter".
Dead-letter is a feature, not a failure
Dead-letter means:
- you stopped retrying
- the event is still visible
- an operator can decide what to do next
This is better than infinite retries (spam) or silent drops (drift).
Delivery logs are what makes all of this usable
Retries without logs create mystery. Logs without retries create drift.
You need both:
- status: pending/delivered/failed/retrying/dead-letter
- attempts count
- last response code/body (or a safe summary)
- timestamps for troubleshooting
Idempotency still matters
Even with bounded retries, receivers must handle duplicates.
Treat webhook payload processing as idempotent by default, because retries are how reliability is achieved.
What to do next
- Implement webhook subscriptions with delivery logs: ERP webhooks (Dubai)
- Keep integration access controlled: API key management (Dubai)