At-least-once is not a bug: receiving webhooks without double-charging

The first time a duplicate webhook charges a customer twice, the architecture question stops being theoretical. Duplicates are not a provider defect: they are the direct consequence of retrying, and retrying is the only way to survive a dropped connection.

When your endpoint does not answer 2xx, the provider assumes the event was lost — even if your handler completed the work and only the response vanished. So it sends again. Any system that promises "exactly once" is either lying or holding state you cannot see.

Order of operations

• Read the raw body. Verify the signature over the exact bytes — a re-serialised JSON object will not match.
• Check the timestamp is recent, to make replay of an old signed payload useless.
• Return 2xx quickly. Do the work after acknowledging, or in a queue. GitHub gives you ten seconds before it counts the delivery as failed.
• Dedupe on the provider's event id inside the same transaction as the work.
• Reconcile on a schedule for anything money-shaped, because retry windows end and events can be missed entirely.

Verify the signature correctly

The common mistake is verifying a JSON.stringify of the parsed body. Whitespace and key order change, the digest changes, and you either reject valid events or, worse, skip verification because it "keeps failing". Capture the raw text first:

import express from 'express';
import { createHmac, timingSafeEqual } from 'node:crypto';

const app = express();
// raw body, not parsed — the digest is over bytes
app.post('/hooks/partner', express.raw({ type: 'application/json' }), (req, res) => {
  const raw = req.body;                       // Buffer
  const sent = req.get('x-signature') || '';
  const timestamp = req.get('x-timestamp') || '';

  if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) return res.sendStatus(400);

  const expected = createHmac('sha256', process.env.PARTNER_SECRET)
    .update(Buffer.concat([Buffer.from(timestamp + '.'), raw]))
    .digest('hex');

  const a = Buffer.from(expected, 'utf8');
  const b = Buffer.from(sent, 'utf8');
  if (a.length !== b.length || !timingSafeEqual(a, b)) return res.sendStatus(401);

  res.sendStatus(202);                        // acknowledge first
  handle(JSON.parse(raw.toString())).catch(() => {});   // then work, idempotently
});

Dedupe where the write happens

An in-memory Set of seen ids survives until the process restarts, which on serverless is measured in minutes. The durable version is a unique constraint, so the duplicate is rejected by the database rather than by your code:

create table webhook_event (
  provider    text not null,
  event_id    text not null,
  received_at timestamptz not null default now(),
  payload     jsonb not null,
  primary key (provider, event_id)
);

-- inside one transaction with the business change:
insert into webhook_event (provider, event_id, payload)
values ('partner', $1, $2)
on conflict do nothing
returning event_id;   -- no row returned means it was already handled

Two properties make this work: the dedupe row is written in the same transaction as the side effect, so a crash cannot leave one without the other; and the key is the provider's stable event id, not a hash of the body — many providers resend the same event with a different envelope.

Ordering is a separate promise

Most providers do not guarantee that event A arrives before event B, even when A happened first. If your handler applies deltas, reordering corrupts state. Two workarounds: carry a version or sequence number on the aggregate and ignore anything older than what you have stored, or store the latest snapshot from the event body rather than applying a diff. "Set status to shipped" is safer than "increment shipped count".

Reconcile what never arrives

Retry windows end. Stripe retries for about three days, Shopify makes eight attempts over four hours, Slack three. An endpoint that is down for a weekend silently loses events, so any integration where a missed event is a business problem needs a periodic pull: list the recent objects from the provider and compare against what you have stored. Treat the webhook as an accelerator and the poll as the truth.

• Verify over raw bytes, with a timestamp window and a constant-time compare.
• Acknowledge before doing the work; keep the handler under the provider's timeout.
• Dedupe with a unique key in the same transaction as the side effect.
• Assume no ordering; use versions or whole-state writes.
• Log every delivery attempt, including the ones your own outbound side makes — "did we send it" is the question support gets first.