Most founders don’t think about their webhook system until something breaks in production. A customer complains their order didn’t process. A Slack alert fires at 2am. You dig into the logs and find thousands of failed deliveries sitting in a queue that was never designed to handle backpressure. The fix is almost always expensive – not because webhooks are hard, but because the original implementation made assumptions that were fine at 50 events per day and catastrophic at 50,000.
We’ve rebuilt webhook infrastructure for a handful of SaaS products at Amora, and the pattern is consistent. The initial build was done fast (fair enough), it worked in staging (of course it did), and then real-world traffic exposed every shortcut. This post is about how to build it properly the first time – or at least understand what “properly” actually means before you commit to an approach.
The Three Failure Modes Nobody Warns You About
Before architecture, you need to understand how webhook systems actually fail in production. Not theoretically – actually.
- Silent delivery failure. You fire a POST, the receiving endpoint returns a 200, and you mark it delivered. But the endpoint timed out before processing the payload, returned 200 anyway to avoid retry spam, and dropped the event on the floor. Your system thinks everything’s fine. It isn’t.
- Retry storms. An external service goes down for 20 minutes. Your retry logic kicks in and hammers it with every queued event simultaneously when it comes back up, causing a second outage. You’ve just become someone else’s DDoS.
- Event ordering assumptions. Your consumer processes webhook B before webhook A because of network latency or queue reordering. If your business logic assumes sequential delivery – and most payment and state-machine flows do – you now have corrupted state that’s almost impossible to detect until a customer screams.
None of these are exotic edge cases. They show up in real products handling real money. Fix the architecture before you hit volume, not after.
The Queue Is Not Optional
If you’re firing webhooks synchronously from within your application’s request cycle, stop. Every webhook delivery that happens inline is latency added to your own system’s response time, and a timeout risk that can cascade into your core product.
The minimum viable webhook architecture for any SaaS that expects growth looks like this: event occurs in your system, an event record is written to a durable queue (not in-memory, not Redis alone – something with persistence guarantees like SQS, Google Pub/Sub, or a Postgres-backed queue like Pgmq or Graphile Worker), a separate worker process reads from that queue and attempts delivery, and the result is recorded against the event.
The queue is the insurance policy. If your worker crashes, events aren’t lost – they’re still in the queue. If the receiving endpoint is down, you can retry later without any manual intervention. If you need to replay events for a customer who had a misconfigured endpoint, you can.
In terms of cost to build this properly from scratch: expect $8,000-$18,000 AUD depending on complexity, whether you need a customer-facing dashboard for webhook management, and how much retry/replay logic you’re building. That’s not a small number, but it’s a fraction of what you’ll spend unpicking a broken implementation six months later.
Retry Logic That Doesn’t Create New Problems
Exponential backoff with jitter. Every senior engineer knows this phrase. Fewer actually implement it correctly.
Here’s what the retry schedule should look like for a production webhook system. First retry after about 30 seconds. Second after 5 minutes. Third after 30 minutes. Then hourly for a few attempts, then daily. You’re trying to smooth out transient failures (network hiccup, brief downtime) without hammering an endpoint that’s genuinely broken.
The jitter part matters more than people think. If you have 10,000 events that all failed at the same moment – say, a receiving service had a 3-minute outage – and your retry logic fires them all at exactly T+30 seconds, you’ve created the retry storm problem from above. Add random variance of 20-30% to each retry interval and you spread the load naturally.
Set a maximum retry count (typically 24-48 hours of attempts) and then dead-letter the event. Don’t retry forever. Customers with permanently broken endpoints shouldn’t be silently consuming resources in your queue indefinitely – and you want a clear signal to surface to them that something’s wrong.
Delivery Guarantees: At-Least-Once Is the Default, and It Has Consequences
This is the part most people skip in their planning, and it creates subtle bugs that are hard to trace.
Virtually every production webhook system delivers at-least-once, not exactly-once. That means retries can cause duplicate deliveries, especially when a POST succeeds on the server side but the response times out before your delivery worker receives it. Your worker doesn’t know the delivery succeeded, so it tries again.
The solution is idempotency keys. Every webhook event you emit should carry a unique, stable event ID. Consumers should store processed event IDs and skip re-processing any event they’ve already handled. This is not your responsibility to enforce on the consumer side – you can’t control what your customers build – but you should document it clearly and provide the ID in a consistent header (X-Webhook-ID or similar).
For your own internal systems that consume webhooks from third parties, always implement idempotency handling. We’ve seen a fintech platform that processed duplicate payment webhooks from a payment gateway during a retry window and double-credited customer accounts. The reconciliation effort took weeks. The fix was two hours of work – it just wasn’t done upfront.
What to Build Into Your Customer-Facing Webhook Management
If you’re building a SaaS that sends webhooks to your customers’ systems – not just consuming them internally – you need a management interface. “Just update the config file” isn’t acceptable for a commercial product. Here’s what that interface needs, roughly in order of priority:
- Endpoint registration and secret management. Customers need to add, edit, and delete their endpoint URLs. Each endpoint should have a unique signing secret so they can verify payloads. Rotate secrets without downtime.
- Event type subscriptions. Don’t send every event type to every endpoint. Let customers subscribe to exactly what they need. This reduces noise and makes their integration cleaner.
- Delivery logs with payload inspection. Show the last N deliveries per endpoint – timestamp, response code, response body, latency. This single feature cuts your support load significantly. Customers can self-diagnose instead of raising tickets.
- Manual retry and replay. Let customers trigger a retry on a failed delivery, or replay any event from the last 30 days. Replay is particularly useful for onboarding: a customer can test their integration against real historical events.
- Delivery failure alerting. Email the customer when an endpoint has failed consistently for X hours. Don’t let them discover a broken integration from their own users.
Building all of this properly – the backend infrastructure, the queue, the retry logic, and the customer-facing dashboard – is a real project. Budget $25,000-$45,000 AUD for a full implementation if you’re starting from zero and building it to production standard. Some teams try to shortcut this with third-party services like Svix or Hookdeck, which can work well at early stages and typically run $200-$800 AUD per month depending on event volume. That’s a legitimate trade-off to make early.
Signature Verification: Don’t Make It Optional
Every webhook payload you send should be signed. Typically this means computing an HMAC-SHA256 of the raw request body using the customer’s endpoint secret, and sending that signature in a header. Your customers’ systems verify the signature before processing the payload.
Why? Because without it, anyone who discovers an endpoint URL can POST arbitrary payloads to it. A malicious actor could trigger payment processing events, user permission changes, or data deletions – whatever your webhook consumer does. This isn’t a theoretical risk.
Make the signing secret long (at least 32 bytes of cryptographic randomness), store it hashed on your side, show it to the customer once on creation, and provide a clear code example for verification in whatever languages your customers commonly use. Python, Node, and Ruby cover most cases. Don’t assume they’ll figure it out from the spec alone.
If you’re not sure where your current webhook implementation sits across these dimensions, or you’re about to scope a new build and want to get the foundations right, talk to Amora about your build before the architecture is locked in. It’s a much easier conversation to have before the code is written than after.
Webhooks are one of those systems that look simple until they’re not. The teams that build them well aren’t smarter – they’ve just learned which shortcuts compound into production incidents. Now you have the list. Don’t ignore it.
Got something you want built?
Amora Digital is an Australian software and AI agency. We scope it, build it, and ship it – live in 28 days. No offshore teams. No surprises.