TL;DR

Most webhook failures that hurt teams are not obvious transport errors. They are false positives where receipt is logged, or 2xx is returned, but the actual business work never completes.
A reliable webhook handler separates receipt from processing. Verify the signature, log the delivery, enqueue the work, and return fast.
Retries need exponential backoff, idempotency, and a dead letter path. Without those, one transient failure becomes silent data loss or duplicate side effects.
The examples use Node.js, Express, Redis, and BullMQ. The model applies to any stack.

Most webhook bugs are not loud. There is no 500, no DNS failure, no alert firing. The provider marks the delivery successful, your route returns 200, and the business action never happens. Nobody knows until a customer asks why their order has not shipped.

The root cause is always the same: receipt and processing are conflated.

This article builds the fix: a Node.js and Express ingress route, a BullMQ worker with exponential backoff and dead lettering, and Redis idempotency with an atomic claim. The model applies to any stack.

Why webhook handlers fail silently

Loud failures like DNS errors, TLS mismatches, and 500s are at least visible. Silent failures are not. They look healthy on the wire and broken in outcomes.

Failure mode	What looks successful	What actually failed
You return success before work finishes	Provider marks delivery OK.	Async work dies later; no durable outcome.
You do heavy work inline	"It worked on my machine."	Timeouts and retries under load.
No retry or dead letter story	Request hit your logs once.	Transient blips become permanent loss.
Receipt and business validation conflate	HTTP path succeeded.	Downstream rejects the event; nobody ties it back.

Delivery confirmation is not processing confirmation.

What the naive handler gets wrong

The first handler most teams ship works in staging and breaks in production. Here is what that typically looks like for an order events endpoint:

server-naive.js

import express from 'express';
 
const app = express();
 
app.post('/webhooks/orders', express.json(), async (req, res) => {
  if (req.get('x-webhook-signature') !== process.env.WEBHOOK_SECRET) {
    return res.status(401).json({ error: 'invalid signature' });
  }
 
  await createInvoice(req.body);
  await sendProvisioningEvent(req.body);
 
  return res.status(200).json({ ok: true });
});
 
app.listen(3000, () => {
  console.log('listening on 3000');
});

Three problems: it verifies against parsed JSON instead of raw bytes; it does real work inline, creating timeout and retry risk; and it treats 200 OK as "processing finished."

If createInvoice or sendProvisioningEvent errors after the response is sent, the provider saw a clean delivery and will not retry. That event is gone.

Flowchart: webhook flows from provider through verification, receipt log, 202 Accepted, and queued job to the worker; dotted paths show a false positive when the provider still sees success, and customer impact that appears later.

How to fix it

Each failure mode above has a direct fix. Applied together, they separate receipt from processing, make retries safe, and give failed deliveries somewhere to land.

State diagram: Received branches to Rejected or Queued; Processing leads to Processed, RetryScheduled (looping back), or DeadLettered; terminal states include Processed, Rejected, and DeadLettered.

1. Verify the signature on raw bytes

Providers sign raw bytes. If you parse the body first, you change the byte representation before you check the signature, which breaks valid events. The route needs express.raw(), not express.json(), for the webhook path.

lib/signature.js

import crypto from 'node:crypto';
 
export function computeSignature(rawBody, secret) {
  return crypto.createHmac('sha256', secret).update(rawBody).digest('hex');
}
 
export function verifySignature(rawBody, providedSignature, secret) {
  const expected = computeSignature(rawBody, secret);
  const expectedBuffer = Buffer.from(expected, 'hex');
  const actualBuffer = Buffer.from(providedSignature ?? '', 'hex');
 
  if (expectedBuffer.length !== actualBuffer.length) return false;
  return crypto.timingSafeEqual(expectedBuffer, actualBuffer);
}

Warning

Do not parse the body before verifying. Changing the byte representation breaks valid signatures.

2. Log receipt before you touch business logic

Receipt needs its own log line before any business logic runs. When debugging a false positive, you need to know exactly when the signed delivery arrived, separate from what happened downstream. A minimal pino logger handles this:

lib/logger.js

import pino from 'pino';
 
export const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  base: undefined,
  timestamp: pino.stdTimeFunctions.isoTime,
});

Each log entry carries a context object that ties the delivery ID, event type, and attempt number together across receipt and every retry:

lib/context.js

export function webhookContext({ deliveryId, eventType, attempt }) {
  return {
    component: 'webhook-handler',
    deliveryId,
    eventType,
    attempt,
  };
}

The three fields (deliveryId, eventType, attempt) stitch logs into a traceable execution trail across receipt, processing, and every retry.

3. Enqueue the work and return 202 immediately

Most providers time out after 10 seconds. That is not enough to safely run business logic under load. Move work off the request thread: verify, log, enqueue, return 202 Accepted. Redis and BullMQ hold the queue.

Step 1: Register the queue and Redis connection

Create the shared Redis connection first, then pass it into the queue.

lib/redis.js

import IORedis from 'ioredis';
 
export const connection = new IORedis(process.env.REDIS_URL ?? 'redis://127.0.0.1:6379', {
  maxRetriesPerRequest: null,
});

With the connection in place, create the queue and configure the retry policy:

jobs/webhook-queue.js

import { Queue } from 'bullmq';
import { connection } from '../lib/redis.js';
 
export const webhookQueue = new Queue('webhook-deliveries', {
  connection,
  defaultJobOptions: {
    attempts: 6,
    backoff: {
      type: 'exponential',
      delay: 1000,
    },
    removeOnComplete: 1000,
    removeOnFail: false,
  },
});

Six attempts with 1s exponential backoff yield about 63s of retry delay before hard failure (1+2+4+8+16+32s).

Step 2: Add the Express route

server.js

import 'dotenv/config';
import express from 'express';
import crypto from 'node:crypto';
import { logger } from './lib/logger.js';
import { verifySignature } from './lib/signature.js';
import { webhookContext } from './lib/context.js';
import { webhookQueue } from './jobs/webhook-queue.js';
 
const app = express();
 
app.get('/health', (_req, res) => {
  res.json({ ok: true });
});
 
app.post('/webhooks/orders', express.raw({ type: 'application/json' }), async (req, res) => {
  const rawBody = req.body;
  const deliveryId = req.get('x-webhook-id') ?? crypto.randomUUID();
  const eventType = req.get('x-webhook-event') ?? 'order.completed';
  const signature = req.get('x-webhook-signature');
 
  if (!verifySignature(rawBody, signature, process.env.WEBHOOK_SECRET)) {
    logger.warn(
      webhookContext({ deliveryId, eventType, attempt: 0 }),
      'webhook signature verification failed'
    );
 
    return res.status(401).json({ error: 'invalid_signature' });
  }
 
  const payload = JSON.parse(rawBody.toString('utf8'));
 
  logger.info(
    webhookContext({ deliveryId, eventType, attempt: 0 }),
    'webhook received'
  );
 
  await webhookQueue.add(
    'process-webhook',
    { deliveryId, eventType, payload },
    { jobId: deliveryId }
  );
 
  return res.status(202).json({ accepted: true, deliveryId });
});
 
const port = Number(process.env.PORT ?? 3000);
app.listen(port, () => {
  logger.info({ port }, 'server listening');
});

202 means accepted for later work, not side effects finished.

4. Make retries idempotent with a delivery ID

Most providers deliver at-least-once. Retries are only safe when they are idempotent. Key idempotency on the delivery ID and use BullMQ's jobId to deduplicate at the queue level before the worker runs. For a deeper look at idempotency, see What Is Idempotent in REST API Design?. The module below manages three states per delivery: unclaimed, processing, and processed:

lib/idempotency.js

import { connection } from './redis.js';
 
const PREFIX = 'processed:webhook:';
const PROCESSED_TTL_SECONDS = 7 * 24 * 60 * 60; // 7 days; aligns with typical provider retry/replay windows
const CLAIM_TTL_SECONDS = 5 * 60; // 5 minutes; should exceed p99 handler time
 
export async function alreadyProcessed(deliveryId) {
  const value = await connection.get(`${PREFIX}${deliveryId}`);
  return value === '1';
}
 
export async function claimDelivery(deliveryId) {
  const key = `${PREFIX}${deliveryId}`;
  const result = await connection.set(key, 'processing', 'EX', CLAIM_TTL_SECONDS, 'NX');
  return result === 'OK';
}
 
export async function releaseClaim(deliveryId) {
  const key = `${PREFIX}${deliveryId}`;
  const value = await connection.get(key);
  if (value === 'processing') {
    await connection.del(key);
  }
}
 
export async function markProcessed(deliveryId) {
  await connection.set(`${PREFIX}${deliveryId}`, '1', 'EX', PROCESSED_TTL_SECONDS);
}

Tip

TTL on the 1 marker caps Redis growth. SET … NX is an atomic claim, so only one worker wins per delivery ID even under concurrent retries.

5. Dead-letter permanent failures

The worker checks idempotency first, runs the business logic, and distinguishes between errors worth retrying and errors that are permanent. A network timeout is worth retrying. A payload that fails business validation will fail every time, so it goes straight to the dead letter queue.

jobs/webhook-worker.js

import { Worker, UnrecoverableError } from 'bullmq';
import { connection } from '../lib/redis.js';
import { logger } from '../lib/logger.js';
import { webhookContext } from '../lib/context.js';
import { alreadyProcessed, claimDelivery, releaseClaim, markProcessed } from '../lib/idempotency.js';
 
async function processWebhook(job) {
  const { deliveryId, eventType, payload } = job.data;
  const ctx = webhookContext({ deliveryId, eventType, attempt: job.attemptsMade });
 
  if (await alreadyProcessed(deliveryId)) {
    logger.info(ctx, 'webhook already processed, skipping');
    return { skipped: true };
  }
 
  const claimed = await claimDelivery(deliveryId);
  if (!claimed) {
    logger.info(ctx, 'webhook claim not acquired, another worker is processing');
    return { skipped: true };
  }
 
  try {
    logger.info(ctx, 'webhook processing started');
 
    if (!payload.orderId) {
      throw new UnrecoverableError('missing orderId, dead lettering');
    }
 
    await simulateBusinessLogic(payload);
 
    await markProcessed(deliveryId);
    logger.info(ctx, 'webhook processed successfully');
 
    return { processed: true };
  } catch (err) {
    await releaseClaim(deliveryId);
 
    if (err instanceof UnrecoverableError) {
      logger.error({ ...ctx, err: err.message }, 'webhook dead lettered due to a permanent error');
      throw err;
    }
 
    logger.warn({ ...ctx, err: err.message, attempt: job.attemptsMade }, 'webhook processing failed, will retry');
    throw err;
  }
}
 
async function simulateBusinessLogic(payload) {
  // Replace with real logic: createInvoice(payload), sendProvisioningEvent(payload), etc.
  if (payload.simulateTransientError) {
    throw new Error('transient error, safe to retry');
  }
  await new Promise((resolve) => setTimeout(resolve, 50));
}
 
export const worker = new Worker('webhook-deliveries', processWebhook, {
  connection,
  concurrency: 5,
});
 
worker.on('failed', (job, err) => {
  const isTerminal = !job || job.attemptsMade >= (job.opts.attempts ?? 1);
  if (isTerminal) {
    logger.error(
      webhookContext({ deliveryId: job?.data?.deliveryId, eventType: job?.data?.eventType, attempt: job?.attemptsMade }),
      `webhook moved to dead letter queue: ${err.message}`
    );
  }
});

Testing the success, retry, and dead-letter paths

The only way to trust a retry and dead-letter path is to trigger it deliberately. Start the server and worker together in one terminal:

Terminal 1

node server.js &
node jobs/webhook-worker.js

Then run the test sender from a second terminal:

scripts/send-test-webhooks.js

import crypto from 'node:crypto';
 
const BASE_URL = process.env.BASE_URL ?? 'http://localhost:3000';
const SECRET = process.env.WEBHOOK_SECRET ?? 'super-secret-demo-key';
 
function sign(body) {
  return crypto.createHmac('sha256', SECRET).update(body).digest('hex');
}
 
async function send(label, payload) {
  const body = JSON.stringify(payload);
  const deliveryId = crypto.randomUUID();
 
  const res = await fetch(`${BASE_URL}/webhooks/orders`, {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      'x-webhook-signature': sign(body),
      'x-webhook-id': deliveryId,
      'x-webhook-event': 'order.completed',
    },
    body,
  });
 
  const json = await res.json();
  console.log(`[${label}] status=${res.status}`, json);
}
 
await send('success', { orderId: 'ord_001', amount: 9900 });
await send('transient', { orderId: 'ord_002', simulateTransientError: true });
await send('dead-letter', { simulateTransientError: false }); // missing orderId

From Terminal 2, run the sender:

Terminal 2

node scripts/send-test-webhooks.js

Watch Terminal 1. The success path logs webhook processed successfully. The transient path logs will retry, then webhook processed successfully after backoff. The dead-letter path logs webhook dead lettered due to a permanent error and stops retrying.

Path	Terminal 1 log	Terminal 2 status
Success	`webhook processed successfully`	`202`
Transient failure	`will retry` then `webhook processed successfully`	`202`
Permanent failure	`webhook dead lettered due to a permanent error`	`202`

When this pattern doesn't fit

A queue-and-worker setup adds real infrastructure: Redis, BullMQ, a persistent worker, and an idempotency store. Use it when the complexity is warranted.

Use it when	Skip it when
Business side effects cannot be lost	Handler is fast and stateless (cache invalidation, lightweight reads)
Providers retry on failure and you need idempotency guarantees	Cold start costs make a persistent worker impractical
Any single handler step takes more than a few hundred milliseconds	Dropped notifications are acceptable by design
You need a dead letter queue to inspect and replay failed deliveries	Managed queues like SQS and Lambda already handle the separation

The goal is always the same: separate receipt from processing, make retries safe, and give failed deliveries somewhere to land. The specific tools are secondary.

Webhook Handlers: Why Yours Fails Silently and How to Fix It