7 webhook integration best practices I learned from failure testing

Reliable webhook integration means treating delivery as at-least-once: verify HMAC signatures on the raw body, retry with backoff, make receivers idempotent on event ID, handle out-of-order updates, and recover or dead-letter what still fails.

I tested each of these through TaskSocial, a task app where I built a webhook sender and a separate receiver and then broke the connection between them on purpose. I used the results to work out what an integrating developer needs from the documentation.

The current experimental flow. Separate database writes and worker coordination mean this is not a production guarantee.

TL;DR

• Treat every delivery as at-least-once and dedupe on a stored event ID.
• Verify the HMAC signature against the raw request body and reject stale timestamps.
• Return a 2xx response quickly and process the event in the background.
• Retry with backoff, and keep exhausted deliveries so they can be replayed.
• Use version checks so older events cannot overwrite newer state.
• Recover deliveries that a crashed worker left mid-attempt.

Assume at-least-once delivery, not exactly-once

Webhook delivery should account for repeated requests. TaskSocial retries a delivery when it does not receive a successful response, up to four attempts. This means the receiver might process an event but the sender might not receive its response. When that happens, the sender can retry an event the receiver has already processed.

Receivers need to recognize previously processed event IDs and handle repeated requests safely. TaskSocial's retry mechanism does not guarantee exactly-once processing or successful delivery in every case. Deliveries that exhaust all four attempts are marked as failed.

Happy-path delivery before you break anything

I first confirmed the basic flow. TaskSocial sent a webhook event, and a separate receiver accepted it. The receiver did not use MongoDB at this stage, so I checked its terminal output to confirm delivery.

With the basic flow working, I moved on to testing what happened when the receiver went offline, a response was lost, or the worker restarted.

How webhook retries and backoff behave in TaskSocial

When I stopped the receiver, TaskSocial recorded fetch failed. The lastResponseStatus field was null because the receiver was offline and TaskSocial never received an HTTP response.

My first retry test exposed a gap. Deliveries marked failed were not attempted again. I changed the worker so that when a delivery attempt failed, it returned to pending if attempts remained. The nextAttemptAt field controlled when the next attempt became eligible to run. Once all four attempts were exhausted, the delivery was marked failed.

The worker was configured for four total delivery attempts: the initial request and up to three retries, with delays of 5, 15, and 30 seconds. It polls for due deliveries every five seconds, so a retry may start slightly later than its configured delay.

Each attempt times out after five seconds, so a slow receiver counts as a failed attempt even if it eventually finishes the work.

// Retry scheduling after an attempt fails.
// Assumes delivery.attempts has already been incremented.
const MAX_ATTEMPTS = 4;
const RETRY_DELAYS = [5, 15, 30]; // seconds

if (delivery.attempts < MAX_ATTEMPTS) {
  const delaySeconds = RETRY_DELAYS[delivery.attempts - 1];

  delivery.status = "pending";
  delivery.nextAttemptAt = new Date(
    Date.now() + delaySeconds * 1000
  );
} else {
  delivery.status = "failed";
}

await delivery.save();

I stopped the receiver, created a task, and started the receiver again. The event eventually arrived. MongoDB showed status: delivered, attempts: 4, lastResponseStatus: 200, and lastError: null.

The delivery record shows delivered, four attempts, HTTP 200, and no recorded error.

This test showed that task creation and webhook delivery are separate steps. A task can be created even when its receiver is unavailable, and retries give temporary delivery failures another chance to succeed.

What happens after all delivery attempts fail?

After four unsuccessful attempts, TaskSocial marks the delivery as failed. The delivery record retains the attempt count, last error, last HTTP response status, event ID, and endpoint URL. An operator can inspect these fields to understand why the delivery failed.

Replaying a failed delivery would require an operation to reset its status to pending, reset or otherwise manage its attempt count, and make it eligible through nextAttemptAt. The current implementation does not provide this operation.

These details help developers integrating with TaskSocial understand when a failed delivery will be retried, how many attempts to expect, and why a webhook might arrive later than expected. Documenting the request timeout and polling interval also helps them troubleshoot delayed or failed deliveries.

The retry test raised another question: what if the receiver processed the event, but the sender never received the response?

Handle duplicate webhook deliveries with durable event IDs

I simulated a lost response after the receiver had processed an event. From TaskSocial's perspective, the request had failed, so it retried. The receiver saw the same event ID twice.

A failed request does not necessarily mean the receiver did nothing. My first solution used a JavaScript Set to remember processed event IDs. It worked until the receiver restarted, because the Set existed only in memory.

I replaced it with MongoDB records and added a unique index on event IDs. After processing an event, I restarted the receiver and sent the same event ID again with a fresh, valid signature. The receiver returned HTTP 200 with "duplicate": true.

// Check whether this event has already been processed.
const existingEvent = await processedEvents.findOne({
  eventId: event.id,
});

if (existingEvent) {
  return res.status(200).json({
    received: true,
    duplicate: true,
  });
}

This snippet shows the lookup and response only. The receiver also validates the request and records processed events in MongoDB.

After restart, the receiver recognizes the same event ID and logs it as a duplicate.

Persistent storage lets the receiver recognize an event ID after a restart, while the unique index prevents duplicate event-ID records. It does not guarantee exactly-once processing. The task-state update and processed-event record are separate writes. If the receiver crashes after updating the task but before recording the event, a retry could process the event again.

The integration guide should explain that delivery may be repeated, that event IDs are stable identifiers for duplicate detection, and that consumers need to handle duplicates. It should not imply that persistent event records make the business operation atomic.

Return 2xx quickly, then process the event

The current receiver processes events inline before returning HTTP 200. A production-oriented approach would be to verify the signature and timestamp, durably record or enqueue the event, and acknowledge it with a 2xx response. A background worker could then handle task updates and other processing. This keeps the response time short and reduces the chance of a timeout causing unnecessary retries.

Verify webhook signatures with HMAC, raw body, and timestamps

I added HMAC-SHA256 signature verification to the receiver. When I sent a request with an invalid signature, the receiver returned HTTP 401.

The forged request receives HTTP 401 Unauthorized.

TaskSocial sends the signature in the X-Webhook-Signature header as sha256=<hex> and the timestamp in the X-Webhook-Timestamp header, and signs the string <timestamp>.<raw request body>.

// Verify the signature against the raw request body.
// Assumes the timestamp and signature format have been validated.
const body = req.body; // raw Buffer

const expected = createHmac("sha256", secret)
  .update(`${timestamp}.`)
  .update(body)
  .digest();

const suppliedHex = signature.slice(7); // after "sha256="

if (!/^[a-f0-9]{64}$/i.test(suppliedHex)) {
  return res.status(401).json({ error: "Invalid signature" });
}

const received = Buffer.from(suppliedHex, "hex");

if (!timingSafeEqual(received, expected)) {
  return res.status(401).json({ error: "Invalid signature" });
}

The receiver calculates its own digest and compares it with the supplied signature using timingSafeEqual. The route uses Express raw-body parsing so verification uses the request bytes rather than a reconstructed JSON object.

I also added a timestamp to the signed data and configured a five-minute validity window. To test expiry, I sent a correctly signed request with a timestamp six minutes old. The receiver returned HTTP 401 with Webhook timestamp expired.

The six-minute-old timestamp is rejected with HTTP 401 and Webhook timestamp expired.

Signature verification checks whether the supplied signature matches the digest calculated from the shared secret, timestamp, and request body. Timestamp validation rejects requests outside the configured window. It does not prevent every replay: a captured request could still be replayed within that window, which is why duplicate-event handling remains important.

The experiment also showed what the documentation needs to explain. Developers need to know the required headers, signature format, how the message is signed, why the raw request body matters, and how timestamps are validated. Simply stating that requests are signed is not enough to implement verification correctly.

Recover deliveries abandoned when a worker dies mid-attempt

Retries handle failed requests, but they do not automatically recover work abandoned while a worker is processing it.

I stopped the TaskSocial worker after it marked a delivery as processing, but before it sent the request. The delivery remained in MongoDB because the worker queried only pending deliveries.

I added a processing timestamp and a 60-second recovery timeout. After that timeout elapsed, I restarted the backend. The worker identified the abandoned delivery, returned it to the retry flow, and sent it to the receiver, which responded with HTTP 200.

[ADD: screenshot or log showing the abandoned delivery being recovered and returning HTTP 200]

This test showed why delivery documentation should explain what happens when a worker stops mid-attempt, not just when an HTTP request fails. Multiple workers could still compete for the same delivery because the implementation does not use an atomic database claim across backend instances. A recovery timeout also needs to account for requests that may still be active.

Handle out-of-order events with version checks

Webhook events can arrive out of order, so a receiver shouldn't assume that the latest request it receives represents the latest state. TaskSocial's task events already included version numbers before this experiment.

In a separate simulation, I delivered version 2 before version 1. Initially, the receiver overwrote the newer title with the older one. I added a version check to the simulated receiver so it would ignore an incoming event if its version was less than or equal to the version already stored. When version 1 arrived after version 2, the receiver kept the newer title.

// Ignore events with an older or already stored version.
if (currentState && event.data.version <= currentState.version) {
  return res.status(200).json({
    received: true,
    stale: true,
  });
}

The code snippet shows only the version check. The full receiver handler also validates the payload and updates the stored task state. The simulation showed how version checks can prevent stale updates, but it did not prove that TaskSocial guarantees event ordering or that concurrent requests will always be handled safely.

Version 1 arrives after version 2 and is ignored as stale, so the newer title stays.

The documentation takeaway is that consumers should not assume events always arrive in the order in which changes occurred. If they use version-based checks, the guide should explain what the version means and how stale events are handled. The TaskSocial experiment showed one possible receiver-side strategy, but it did not establish a sender-side ordering guarantee.

What webhook docs must spell out for integrators

The experiments changed what I considered necessary in the TaskSocial webhook guide. The endpoint and payload explained how to receive an event, but developers also needed to understand what a failed delivery meant and why the same event could arrive again.

The lost-response test became this guidance in the guide’s “Retries and timeouts” section:

A timeout does not prove that the receiver did no work. The receiver may have processed the event, but its response might not have reached the sender. Consequently, the sender may deliver the same event more than once.

This connects an observable failure to a consumer responsibility: the receiver must handle repeated event IDs rather than assume each request represents a new event.

I also documented the five-second request timeout, four total delivery attempts, configured retry delays, and polling interval. The guide covers signature verification, persistent duplicate detection, and version-based handling of stale updates, while explaining the limitations of each.

The remaining gaps matter too. TaskSocial saves a task and queues its webhook in separate database operations, so a task can be saved even if queuing fails. Event creation and delivery-record creation are separate writes, so an event can exist without a delivery record if the subsequent write fails. The receiver's task-state update and processed-event record are also separate operations. Finally, the worker recovery mechanism does not guarantee exclusive handling across multiple backend instances.

The TaskSocial webhook guide lists these gaps, along with the missing dead-letter replay, so integrators know what can still fail.