Debugging HMAC signature mismatches

You have a webhook endpoint that returns 401 on every delivery, the provider insists it is signing correctly, and you are certain the secret is right. This is the single most common failure in HMAC Signature Verification, and it is almost never a cryptography problem — HMAC-SHA256 is deterministic, so two different digests mean two different inputs. Something between the provider’s signing call and your createHmac call changed a byte, an encoding, or a header. This page is a diagnostic procedure for finding which one, in the order that finds it fastest. If you are still building the verifier rather than repairing it, start from Step-by-step HMAC webhook validation in Node.js; if you are not sure the provider is even using a shared secret, HMAC-SHA256 vs RSA asymmetric webhook signatures explains what an asymmetric header looks like instead.

The discipline that makes this tractable is refusing to guess. Every candidate cause below is a testable proposition about the bytes, and each one can be confirmed or eliminated from a single captured delivery without asking the provider to resend anything. Work the checks in cost order: the ones that need no provider cooperation and no redeploy come first.

Prerequisites

• Runtime: Node.js 20+ with TypeScript 5+ and node:crypto.
• One captured failing delivery — the raw request buffer written to disk, plus the complete header set. Everything below is offline analysis on that fixture; you should not need the provider to resend.
• The provider’s documented test vector: a fixed body, a fixed secret, and the digest they expect. Most providers publish one. Without it you are debugging two unknowns at once.
• Access to the origin (a port-forward, a bastion, or a staging host on the same ingress path) so you can send a request that bypasses the CDN and WAF.
• A working single-secret verifier to repair. If you are also mid-rotation, confirm the overlap window is open first — see Zero-downtime webhook secret rotation, because a retired secret produces exactly the same symptom.

Step 1: Capture the exact bytes your verifier hashed

Nothing else matters until you can prove what bytes reached createHmac. Frameworks are the usual culprit: express.json(), Fastify’s default JSON parser, Next.js route handlers reading await req.json(), and API gateways with body transformation all hand you a reconstructed document, not the wire bytes. Re-serialising a parsed object rewrites key order, drops insignificant whitespace, normalises \uXXXX escapes into raw UTF-8, and re-encodes numbers — all of which change the digest while leaving the JSON semantically identical.

Register a raw-body capture ahead of every parser and log a fingerprint of the buffer rather than the buffer itself.

import express from 'express';
import crypto from 'node:crypto';
import { writeFileSync } from 'node:fs';

const app = express();

// Must be registered BEFORE any JSON body parser on this path.
app.use('/webhooks', express.raw({ type: '*/*', limit: '2mb' }));

app.post('/webhooks/provider', (req, res) => {
  const raw = req.body as Buffer;

  // Persist the fixture so the rest of the diagnosis happens offline.
  writeFileSync('/tmp/webhook-capture.bin', raw);

  console.log(
    JSON.stringify({
      evt: 'webhook_raw_capture',
      byteLength: raw.length,
      bodySha256: crypto.createHash('sha256').update(raw).digest('hex'),
      firstBytesHex: raw.subarray(0, 16).toString('hex'),
      lastBytesHex: raw.subarray(Math.max(0, raw.length - 4)).toString('hex'),
      contentType: req.headers['content-type'] ?? null,
      contentLength: req.headers['content-length'] ?? null,
      headerNames: Object.keys(req.headers).sort(),
    }),
  );

  res.status(200).end();
});

app.listen(3000);

Two fields decide the case immediately. If byteLength differs from the Content-Length header, something rewrote the body in transit or in your stack. If lastBytesHex ends in 0a, the payload carries a trailing newline that the provider may or may not have signed.

Step 2: Print every candidate digest at once

Once you hold the raw buffer, stop testing hypotheses one at a time. The plausible causes form a small cross-product: a handful of body variants, two or three ways of decoding the secret, and two digest encodings. Twenty candidates cost microseconds to compute, and exactly one of them will equal the header value — that one names your bug.

import crypto from 'node:crypto';
import { readFileSync } from 'node:fs';

export type Candidate = { readonly label: string; readonly digest: string };

function stripTrailingLf(buf: Buffer): Buffer {
  const end = buf.length > 0 && buf[buf.length - 1] === 0x0a ? buf.length - 1 : buf.length;
  return buf.subarray(0, end);
}

function reserialise(buf: Buffer): Buffer {
  try {
    return Buffer.from(JSON.stringify(JSON.parse(buf.toString('utf8'))), 'utf8');
  } catch {
    return buf; // Not JSON; the variant is simply not applicable.
  }
}

function prefixed(prefix: string, buf: Buffer): Buffer {
  return Buffer.concat([Buffer.from(prefix, 'utf8'), buf]);
}

export function candidateDigests(raw: Buffer, secret: string, timestamp: string): Candidate[] {
  const bodies: Array<[string, Buffer]> = [
    ['raw bytes', raw],
    ['raw minus trailing LF', stripTrailingLf(raw)],
    ['re-serialised JSON', reserialise(raw)],
    ['latin-1 re-encoded', Buffer.from(raw.toString('utf8'), 'latin1')],
    ['timestamp + "." + raw', prefixed(`${timestamp}.`, raw)],
    ['v1:timestamp: + raw', prefixed(`v1:${timestamp}:`, raw)],
  ];

  const keys: Array<[string, Buffer]> = [
    ['secret as utf-8', Buffer.from(secret, 'utf8')],
    ['secret hex-decoded', Buffer.from(secret, 'hex')],
    ['secret base64-decoded', Buffer.from(secret, 'base64')],
  ];

  const out: Candidate[] = [];
  for (const [bodyLabel, body] of bodies) {
    for (const [keyLabel, key] of keys) {
      for (const encoding of ['hex', 'base64'] as const) {
        out.push({
          label: `${bodyLabel} | ${keyLabel} | ${encoding}`,
          digest: crypto.createHmac('sha256', key).update(body).digest(encoding),
        });
      }
    }
  }
  return out;
}

export function findMatch(headerValue: string, candidates: Candidate[]): Candidate | undefined {
  // Providers prefix the digest: "sha256=abc", "v1=abc", or "t=...,v1=abc".
  const bare = headerValue.split(',').pop()!.replace(/^\s*(sha256|sha1|v1|v0)=/i, '').trim();
  return candidates.find((c) => c.digest === bare);
}

// Run: npx tsx candidates.ts
const raw = readFileSync('/tmp/webhook-capture.bin');
const header = process.env.CAPTURED_SIGNATURE!;
const list = candidateDigests(raw, process.env.WEBHOOK_SECRET!, process.env.CAPTURED_TIMESTAMP ?? '');
for (const c of list) console.log(c.digest.slice(0, 16).padEnd(18), c.label);
console.log('MATCH:', findMatch(header, list)?.label ?? 'none — widen the variant set');

The latin-1 re-encoded variant catches the charset class of bug: a sender that built the signature over Buffer.from(body, 'latin1') or a Java String.getBytes() call with a platform default charset produces a different digest for any payload containing a non-ASCII character, and only for those payloads. That is why the failure often looks intermittent — it tracks customer names and currency symbols, not load.

Step 3: Bisect the signed-string template

If no candidate matched, the signed string is assembled differently from anything you guessed. Bisect it against the provider’s published test vector instead of against live traffic — a fixed body and a fixed secret with a known expected digest removes every variable except the template itself.

Templates in the wild differ along four axes: the version tag (v1, v0, none), the separator (., :, \n, none), the timestamp position (prefix, suffix, absent), and whether extra fields such as the HTTP method, the request path, or the webhook-id header join the string. Enumerate the axes rather than reading the documentation a fourth time.

import crypto from 'node:crypto';

type TemplatePart = 'tag' | 'timestamp' | 'method' | 'path' | 'body';

interface Vector {
  readonly secret: string;
  readonly body: string;
  readonly timestamp: string;
  readonly method: string;
  readonly path: string;
  readonly expected: string; // the digest the provider documents
}

const TAGS = ['', 'v1', 'v0'];
const SEPARATORS = ['', '.', ':', '\n', '|'];
const ORDERS: TemplatePart[][] = [
  ['body'],
  ['timestamp', 'body'],
  ['tag', 'timestamp', 'body'],
  ['method', 'path', 'timestamp', 'body'],
  ['body', 'timestamp'],
];

function render(order: TemplatePart[], tag: string, sep: string, v: Vector): string {
  const value = (part: TemplatePart): string =>
    part === 'tag' ? tag
      : part === 'timestamp' ? v.timestamp
      : part === 'method' ? v.method
      : part === 'path' ? v.path
      : v.body;
  return order.map(value).join(sep);
}

export function bisectTemplate(v: Vector): string | null {
  for (const order of ORDERS) {
    for (const tag of TAGS) {
      if (!order.includes('tag') && tag !== '') continue; // avoid duplicate work
      for (const sep of SEPARATORS) {
        const signed = render(order, tag, sep, v);
        const digest = crypto.createHmac('sha256', v.secret).update(signed, 'utf8').digest('hex');
        if (digest === v.expected) {
          return `order=[${order.join(',')}] tag="${tag}" sep=${JSON.stringify(sep)}`;
        }
      }
    }
  }
  return null;
}

Feed it the vector straight from the provider’s docs. When bisectTemplate returns a description, you have the exact construction to implement; when it returns null, the signed string includes a field you have not modelled — add it to TemplatePart and rerun rather than reverting to trial and error in production.

Engineering Note: Keep this bisector as a permanent test fixture, not a scratch script. Re-running it against the provider’s vector in CI turns a silent template change on their side into a failing build instead of a production outage.

Step 4: Trace the signature header across every proxy hop

A digest you cannot compute is one problem; a header you never received is another, and they present identically as 401. Node lowercases incoming header names, so casing differences between X-Webhook-Signature and x-webhook-signature never matter at the application. What does matter is whether an intermediary removed the header entirely.

The usual offenders: nginx drops headers containing underscores unless underscores_in_headers on is set; AWS API Gateway and ALB drop headers that are not in an explicit mapping when request transformation is configured; CDNs and WAFs forward an allowlist and silently discard the rest; and some ingress controllers strip anything they classify as non-standard.

import type { Request } from 'express';

const KNOWN_SIGNATURE_HEADERS = [
  'x-webhook-signature',
  'x-hub-signature-256',
  'webhook-signature',
  'x-signature-256',
  'signature',
] as const;

export function locateSignatureHeader(req: Request): { name: string; value: string } | null {
  for (const name of KNOWN_SIGNATURE_HEADERS) {
    const value = req.headers[name];
    if (typeof value === 'string' && value.length > 0) return { name, value };
  }
  return null;
}

export function headerDiagnostics(req: Request): Record<string, unknown> {
  const names = Object.keys(req.headers);
  return {
    // Never log the value itself — a signature plus a body is a replayable pair.
    signatureHeaderFound: locateSignatureHeader(req) !== null,
    signatureLikeNames: names.filter((n) => /sign|hmac|digest|hub/i.test(n)),
    hopHeaders: names.filter((n) => /^(via|x-forwarded-|cf-|x-amzn-)/i.test(n)),
    totalHeaderCount: names.length,
  };
}

If signatureLikeNames is empty at the application but the provider’s delivery log shows the header being sent, the loss happened in front of you. Confirm by sending the captured fixture directly to the origin, bypassing every proxy:

# Replay the captured delivery straight at the origin, skipping CDN and WAF.
curl -sS -o /dev/null -w 'origin=%{http_code}\n' \
  --resolve api.yourdomain.com:443:10.0.4.17 \
  -H "Content-Type: application/json" \
  -H "X-Webhook-Timestamp: $CAPTURED_TIMESTAMP" \
  -H "X-Webhook-Signature: $CAPTURED_SIGNATURE" \
  --data-binary @/tmp/webhook-capture.bin \
  https://api.yourdomain.com/webhooks/provider

--data-binary is mandatory. Plain -d strips newlines and re-wraps the payload, which changes the bytes and reproduces the very bug you are chasing.

Verification and testing

Turn each diagnosis into a regression test so the same class of bug cannot return silently.

• Byte-identity assertion. In an integration test, post a fixture containing a non-ASCII character, a trailing newline, and out-of-order keys, then assert the verifier’s captured buffer is Buffer.compare(captured, fixture) === 0. This fails the moment somebody registers a body parser above the route.
• Provider vector test. Pin the provider’s documented body, secret, timestamp, and expected digest into a unit test and assert your production computeSignature reproduces it exactly.
• Encoding negative test. Assert that a valid hex digest presented as base64 is rejected, proving your comparison is not accidentally lenient about format.
• Header-presence test. Assert a request with no signature header returns 401 with a distinct error code (ERR_SIGNATURE_HEADER_MISSING) from a request with a wrong signature (ERR_SIGNATURE_MISMATCH). Two codes make the proxy case visible in dashboards without a code change.

import crypto from 'node:crypto';
import { computeSignature } from './verify';

const VECTOR = {
  secret: 'whsec_test_key',
  body: '{"id":"evt_1","amount":1000}',
  timestamp: '1750000000',
  expected: crypto
    .createHmac('sha256', 'whsec_test_key')
    .update('1750000000.{"id":"evt_1","amount":1000}', 'utf8')
    .digest('hex'),
};

test('production signer reproduces the provider vector', () => {
  const actual = computeSignature(Buffer.from(VECTOR.body, 'utf8'), VECTOR.secret, VECTOR.timestamp);
  expect(actual).toBe(VECTOR.expected);
});

test('non-ascii payload survives the raw capture unchanged', () => {
  const fixture = Buffer.from('{"note":"café"}\n', 'utf8');
  expect(fixture.length).toBe(17); // 15 visible chars + 2-byte é + LF
});

Emit signature_verification_failed with a reason label rather than a bare counter. A dashboard split by missing_header, length_mismatch, digest_mismatch, and timestamp_expired turns the next incident into a glance instead of a repeat of this whole procedure. That labelling is the same signal used for alerting on webhook delivery failures.

Failure modes and gotchas

• Logging the signature next to the body. The instinct when debugging is to dump everything. A log line containing both the raw payload and its valid signature is a ready-made replay attack for anyone with log access, and it survives long after the incident. Log a SHA-256 fingerprint of the body and the length of the signature, never the signature itself; if you must capture a fixture, write it to a short-lived file with restricted permissions rather than to your log pipeline. Pair this with nonce-based replay protection with Redis so a leaked pair cannot be reused even once.
• “Fixing” the mismatch by relaxing the check. Under outage pressure someone will propose skipping verification for one provider, comparing only the first N characters, or falling back to === after timingSafeEqual throws. Each turns an availability incident into a permanent authentication hole, and none of them get reverted. If you must ship before the diagnosis lands, accept the delivery into a quarantine queue unprocessed — do not accept it as verified.
• Debugging against live traffic instead of a fixture. Every retry the provider sends carries a new timestamp and a new signature, so you are chasing a moving target and cannot tell whether a change helped. Capture one delivery, freeze it, and iterate offline. The entire candidate cross-product runs in under a millisecond against a fixture.
• Assuming the secret is per-integration when it is per-tenant. In multi-tenant setups the correct secret depends on a tenant id carried in the path, a header, or the payload itself. Selecting the wrong tenant’s secret produces a perfectly formed digest that simply does not match, indistinguishable from every other cause above. Log which secret identifier was selected (never its value) and assert it matches the delivery’s tenant before you suspect anything cryptographic.
• Trailing newlines introduced by your own tooling. A payload replayed with curl -d @file, copied through an editor that enforces a final newline, or piped through echo instead of printf gains a byte that the original never had. Always use --data-binary and check lastBytesHex on both the original capture and the replay.

Frequently Asked Questions

Related

• Step-by-step HMAC webhook validation in Node.js
• HMAC-SHA256 vs RSA asymmetric webhook signatures
• HMAC Signature Verification
• Zero-downtime webhook secret rotation
• Debugging failed webhook deliveries