Reliable webhooks: no missed events, no double processing

Your app takes webhooks from Stripe, Shopify and your CRM, and now and then an order stays unpaid or a customer gets the same email twice. The providers document why: events can arrive late, twice or out of order, and Shopify says delivery isn't always guaranteed. A reliable endpoint expects all of it.

Why do events go missing or run twice?

Mostly because of retries. If your endpoint errors, times out or replies with anything other than a 2xx status, the provider sends the event again. Stripe and Shopify count a redirect as a failure too, so an endpoint URL that redirects fails on every delivery.

• Stripe retries live events for up to three days, with exponential backoff.
• Shopify retries 8 times over 4 hours; after 8 consecutive failures it deletes a subscription created through the Admin API.
• HubSpot retries up to 10 times over 24 hours.

A handler that replies late runs the event twice; one that keeps failing runs out of retries and misses the change.

How do you know a webhook is genuine?

Check the signature on every request, against the raw body, before trusting any field. Anyone can post to the URL, and Stripe warns that a fake event could fulfil an order or grant account access.

• Stripe: the Stripe-Signature header and the endpoint's whsec_ secret, checked with Stripe's library, which also rejects stale timestamps to stop replays.
• Shopify: a base64 HMAC-SHA256 in X-Shopify-Hmac-SHA256, computed from the raw body with your app's client secret.
• HubSpot: X-HubSpot-Signature, built from your app secret and the un-parsed body.

Stripe says any change to the body makes verification fail, and Shopify says to verify before any body-parsing middleware runs. Reject a failed check with a 4xx and log it.

What should the endpoint do before it replies?

Verify, save, reply, nothing more; the real work runs later in a background job. Shopify allows five seconds for the whole request and HubSpot retries replies slower than that. Stripe gives no figure, but says to return a 2xx before any complex logic and to use an asynchronous queue, so spikes such as monthly renewals don't overwhelm your servers.

// app/api/webhooks/stripe/route.ts
export async function POST(req: Request) {
  const body = await req.text() // raw body, unparsed
  let event
  try {
    event = stripe.webhooks.constructEvent(body,
      req.headers.get('stripe-signature') ?? '', process.env.STRIPE_WEBHOOK_SECRET!)
  } catch {
    return new Response('Invalid signature', { status: 400 })
  }
  // unique key on (source, eventId): a repeat inserts nothing
  await db.webhookEvents.insertIgnoreDuplicate({
    source: 'stripe', eventId: event.id, type: event.type, payload: body,
  })
  return new Response(null, { status: 200 })
}

The table doubles as the queue: a worker processes new rows and marks each done or failed, so a crash leaves the event waiting, not lost.

How do you stop the same event running twice?

Store each event id under a unique key, skip any id you have seen, and still reply with a 2xx so the retries stop.

• Stripe: log processed event ids. In some cases it sends two separate Event objects, so also check the data.object id with the event type.
• Shopify: dedupe on X-Shopify-Webhook-Id. Two subscriptions to one topic get different webhook ids that share one X-Shopify-Event-Id.
• HubSpot: a request can carry a batch, a notification can repeat and eventId isn't guaranteed unique, so key on more than one field.

Then make the work safe to repeat, as Shopify recommends: set an order to paid rather than adding a payment, and update an existing record instead of creating another.

What if events arrive out of order?

Never depend on order, because neither Stripe nor Shopify guarantees it. A new Stripe subscription can deliver invoice.paid before customer.subscription.created, and a Shopify products/update can land before products/create. Don't sort Stripe events by created either: it is recorded in seconds.

• Compare times before you overwrite: Shopify recommends X-Shopify-Triggered-At or updated_at, and HubSpot sends occurredAt. Ignore anything older than what you applied.
• Or treat the event as a prompt and fetch the current object from the API. Stripe suggests this, and its thin events, recommended for new integrations, include a method for it.

How do you find and replay failed events?

Log every delivery with its source, id, type, status and error, and alert a named person when events fail or sit unprocessed. Stripe says signature failures most often mean the wrong signing secret. Then recover with each provider's tools.

• Stripe: Workbench's Event deliveries tab marks events Delivered, Pending or Failed. Dashboard Resend works for 15 days after an event is created and stripe events resend in the CLI for 30. A resend doesn't cancel automatic retries, so dedupe matters here too.
• Shopify: for Dev Dashboard and CLI apps, the Logs page shows the past 7 days, and warning emails go to the app's emergency developer address. To recover, Shopify suggests fetching the missed period through its API and running a regular reconciliation job.

Where to start

Check each endpoint, starting with the one that moves money:

• Signature checked against the raw body; failures rejected and logged.
• A 2xx within seconds, with slow work in a background job.
• Event ids under a unique key, repeats skipped with a 2xx, and writes safe to repeat.
• No assumption about order; current state fetched before anything critical.
• Failures alerted to a named person, with a replay plan for each provider.

Our web app development service builds webhook endpoints this way into Next.js products. If events are already going missing, send us the details and we will trace where they drop.