Debugging Failed Webhook Deliveries

A delivery marked “failed” in a provider’s dashboard tells you almost nothing on its own — it could be a signature mismatch, a handler timeout, or a 5xx from a dependency three layers down. This page extends inspecting and replaying webhook deliveries with a disciplined triage that turns that vague signal into a root cause, and it pairs naturally with testing webhooks locally with ngrok and tunnels so you can reproduce the exact failing request against a debugger on your laptop. The method is always the same: read the captured delivery, classify the failure, recompute what should have happened, and replay until it reproduces.

Prerequisites

• A delivery log that stores the raw body bytes, all headers, the response code, and a correlation ID (see the capture patterns in the parent guide).
• The shared signing secret and knowledge of the provider’s HMAC scheme.
• The ability to replay a stored delivery idempotently against a local or staging handler.
• curl and a Python REPL for ad-hoc signature recomputation.
• For local reproduction, a tunnel so the provider (or your replay tool) can reach your machine.

Reading the Response Code as Evidence

Before touching a debugger, treat the status code the provider recorded as the first piece of physical evidence. It is weak evidence — plenty of stacks return 500 for anything unhandled — but it narrows the search from “the integration is broken” to one of four or five concrete hypotheses, and each hypothesis has a different first artefact to pull. The table below is the mapping worth memorising, because it is what stops an on-call engineer from rotating a signing secret at 03:00 when the real problem was a saturated database connection pool.

Two patterns in that table are worth calling out because they are routinely misdiagnosed. First, a 401 that appears for every delivery starting at a precise minute is a secret-rotation bug, not a signing bug; a 401 scattered across a few percent of deliveries is almost always body mutation on a specific content path (a proxy that recompresses, a middleware that only re-serializes when a charset is present). Second, 500 responses that correlate with retry attempt numbers greater than one are usually duplicate-key violations from a non-idempotent handler rather than genuine defects — the first attempt succeeded, the provider never saw the acknowledgment, and the retry collided with the row the first attempt wrote.

Record which hypothesis you are testing before you start. Webhook debugging drifts easily: you pull a delivery to check a signature, notice an unrelated warning in the log, and forty minutes later you are reading a connection-pool patch with the original failure still unexplained. Writing “hypothesis: body mutated by the gzip middleware” in the incident channel takes ten seconds and makes the next step falsifiable.

Step-by-Step Implementation

1. Pull the failed delivery

Query the log for the specific failure rather than eyeballing a stream. Anchor on the response code and a time window.

psql -c "SELECT correlation_id, response_code, received_at
         FROM webhook_deliveries
         WHERE response_code >= 400
         ORDER BY received_at DESC LIMIT 20;"

Widen that query in one direction at a time. Grouping by response_code, event_type over the failing hour tells you whether the fault is scoped to one event type — which points at a payload change — or spread across all of them, which points at infrastructure. Grouping by the provider’s attempt header separates first-attempt failures (a real defect) from retry-only failures (usually idempotency collisions). If the log holds the source IP, group by that too: a single provider egress IP failing while others succeed is a symptom of one of their dispatchers running an older signing implementation, and it is worth reporting to them rather than debugging on your side.

2. Classify the failure

The response code is your first branch. A 401/403 is almost always a signature problem. A handler that logs 408/499 or a provider that reports “timeout” points at slow processing — if those bunch around a traffic peak, reproduce the peak first by simulating webhook traffic spikes rather than chasing an individual request. A 500/502/503 means your handler accepted the request but threw or a dependency was down. Classify before you change anything — fixing the wrong branch wastes the incident window.

The strongest classification signal is not the code itself but its shape over time. Plot failures per minute for the affected event type and read the edge: a vertical step from zero to a steady rate means a discrete change — a deploy, a secret rotation, a provider release — and you should be reading a changelog, not a stack trace. A gradual ramp that tracks traffic volume means a capacity limit, and the fix lives in worker counts, pool sizes, or the backpressure path rather than in the handler. A sawtooth that recovers every few minutes usually means a dependency with its own retry or circuit-breaker cycle underneath you. Fifteen seconds looking at that shape routinely saves an hour of reading code that was never broken.

3. Recompute the signature on the raw bytes

For a suspected signature mismatch, recompute the HMAC over the exact stored bytes and compare to the header the provider sent. A difference between recomputed-from-raw and recomputed-from-reparsed proves the body was mutated before verification — the single most common signing bug.

import hmac, hashlib, json, psycopg2

conn = psycopg2.connect("dbname=webhooks")
cur = conn.cursor()
cur.execute("SELECT raw_body, headers FROM webhook_deliveries WHERE correlation_id=%s",
            ("abc123def456",))
raw, headers = cur.fetchone()
raw = bytes(raw)
secret = b"whsec_..."

ts = json.loads(headers)["X-Webhook-Timestamp"]
expected = hmac.new(secret, f"{ts}.".encode() + raw, hashlib.sha256).hexdigest()
received = json.loads(headers)["X-Webhook-Signature"].split("v1=")[-1]

print("from raw bytes:", hmac.compare_digest(expected, received))   # True => signing OK
# If False, recompute after re-serializing to confirm a body-mutation bug:
reparsed = json.dumps(json.loads(raw)).encode()
print("from reparsed :",
      hmac.compare_digest(hmac.new(secret, f"{ts}.".encode()+reparsed,
                                   hashlib.sha256).hexdigest(), received))

Read the two booleans as a truth table. True from the raw bytes means signing was fine and the 401 came from somewhere else — most often a timestamp outside the tolerance window, which you confirm by subtracting the header timestamp from received_at. False from raw but True from reparsed is the definitive proof of body mutation: something between the socket and your verifier normalised the JSON, and the fix is to move signature verification earlier in the middleware chain rather than to change the hashing. False from both means either the wrong key or a different signed-message construction than you assumed — check whether the provider signs timestamp.body, the body alone, or a canonical string that includes the method and path, and check whether the digest is hex or base64 before concluding the secret is wrong.

If both are False and you have more than one secret in play, brute-force the small space rather than reasoning about it: loop the recomputation over every active and recently retired secret and print which one matches. A match against a retired secret is a rotation that overlapped a queued delivery, and it argues for a longer dual-accept window in your key rotation procedure rather than for any change to the handler. Byte-level surprises are also worth ruling out early — a trailing newline, a UTF-8 BOM, or a body that was gzip-decoded before capture all change the digest while looking identical in a log viewer, so compare len(raw) against the Content-Length header the provider sent before you trust anything else.

4. Reproduce locally with a replay

Start your handler locally behind a tunnel, then replay the captured raw delivery into it so you can attach a debugger to the exact failing input. Re-sign with a fresh timestamp so your own replay-attack prevention window does not reject it.

TS=$(date +%s)
SIG=$(python -c "import hmac,hashlib,sys; \
  print(hmac.new(b'whsec_...', f'$TS.'.encode()+open('body.bin','rb').read(), \
  hashlib.sha256).hexdigest())")
curl -i -X POST http://localhost:8000/webhooks/orders \
  -H "Content-Type: application/json" \
  -H "X-Webhook-Timestamp: $TS" \
  -H "X-Webhook-Signature: t=$TS,v1=$SIG" \
  --data-binary @body.bin

5. Fix and re-verify against the corpus

Apply the fix, then replay the whole captured corpus of recent deliveries — not just the one — to confirm you resolved the class of failure without regressing others. If the corpus is large enough to be worth keeping, promote it into your webhook mocking and sandbox environments so every future build replays the same traffic automatically.

Size the corpus deliberately. A few hundred deliveries spanning every event type you handle, plus every distinct failure you have ever debugged, runs in well under a minute and catches the overwhelming majority of regressions; ten thousand deliveries of the same three event types take twenty times as long and catch nothing extra. Stratify instead of sampling uniformly: keep every distinct (event_type, schema_version) combination, every payload that has ever produced a non-2xx response, and a handful of ordinary successes per type as controls. When the corpus finds nothing for several months, that is a signal to re-record it against recent traffic rather than a signal that it is working.

Roll the fix out in the order that keeps blast radius small: deploy behind a flag or to a single replica first, replay ten deliveries from the failing window at it, confirm the class is resolved, then release the provider’s own retries by re-enabling the endpoint or triggering a bulk replay. Reversing that order — restoring traffic first and validating afterwards — turns a contained failure into a second incident when the fix is wrong, because the provider will have retried a full backlog against it.

Verification and Testing

Prove the fix with an assertion, not a glance. After replaying the corpus, the handler log must show zero failures for the previously failing event type:

# Replay corpus, then assert no failures remain.
python replay_corpus.py --since "1 hour ago"
test "$(grep -c '"webhook_failed"' handler.log)" -eq 0 && echo "PASS" || echo "FAIL"

For the signature case specifically, the unit test should feed the stored raw bytes through your verification function and assert it now returns true. Persisting these reproductions as fixtures means the next deploy runs them automatically — turning a one-off debug into a permanent regression guard.

Assert on more than the absence of errors, because a handler that silently skips work also produces zero failures. The useful post-fix assertions are three: the replay produced the expected side effect exactly once (query the target row and check its version or updated timestamp), the idempotency store recorded exactly one claim per event ID, and the handler’s own success counter incremented by the number of deliveries you replayed. A replay that returns 200 on every request while the side-effect count stays flat means your idempotency guard is swallowing the corpus — correct behaviour for production, useless for verification, which is why the corpus should run against a scratch database seeded fresh for each run.

Keep one delivery from each debugged failure as a named test case rather than folding it anonymously into the corpus. Name it after the bug (test_replay_rejects_body_with_bom, test_retry_of_processed_event_returns_200) so the next engineer who breaks it reads a description of the original failure instead of a correlation ID. That naming discipline is the difference between a regression suite people trust and one they delete when it goes red during a release.

Failure Modes and Gotchas

• Body mutated before verification. A middleware that parses and re-serializes JSON (or strips a trailing newline) invalidates the signature. Always verify against the untouched raw bytes captured at ingress.
• Timeout from synchronous work. If the handler does the real processing before responding, slow dependencies cause provider timeouts and retries. Acknowledge fast and defer via the synchronous versus async webhook split, then debug the worker separately.
• 5xx that is really a duplicate. A retried delivery hitting a non-idempotent handler can 500 on a unique-constraint violation. Confirm the event ID against your idempotency store before assuming a code bug.
• Stale-timestamp rejection on replay. Re-sending the original headers trips the replay window. Re-sign with a current timestamp for trusted internal replays.
• Debugging the load balancer’s timeout, not the handler’s. If the proxy in front of your service closes the connection at 30 s while your handler is happy to run for 60 s, the provider records a 504 and your application log shows a successful request. Compare the two timeout values before you profile a single line of handler code.
• A reproduction that quietly skips verification. Posting the stored body at a local handler with signature checking disabled “just to see the parse error” reproduces a different program. Keep verification on and re-sign; if the local secret differs from production’s, load the production development secret rather than turning the check off.
• Chasing a failure the provider already succeeded on. Most delivery logs show the final state of an event, not each attempt. A delivery whose attempt 1 failed and attempt 3 succeeded needs no fix at all beyond confirming idempotency held; filter to events whose terminal state is failed before opening an investigation.

Keeping the Evidence Debuggable Without Keeping It Forever

Everything in this method depends on the captured delivery still existing when you need it, and the interval between a delivery failing and someone noticing is longer than teams expect. A customer reports a missing order the next morning; a reconciliation job finds a gap at the end of the month. If your capture table holds twenty-four hours, both of those investigations start with nothing. At the same time, raw webhook bodies are among the most sensitive data in the system — they contain customer names, addresses, amounts, and sometimes tokens — so “keep everything forever” is not available either.

The resolution is a tiered lifecycle in which debuggability decays deliberately. Keep the full raw body for long enough to cover the realistic detection lag — seven days is a good default, fourteen if your reconciliation runs weekly. After that, strip the body to a redacted form that preserves structure and identifiers but drops free-text and financial fields; a redacted body still supports schema-drift diffs and event-type analysis, which is what most late investigations actually need. After thirty days, drop to metadata only: correlation ID, event type, response code, timings, attempt number. That tier costs almost nothing per row and still answers “how often did this happen last quarter”. Purge at ninety days unless a compliance regime says otherwise.

Run the tier transitions as a scheduled job with a dry-run mode, and alert if it has not run — a silently stalled redaction job is a data-protection incident waiting to be discovered by an auditor rather than by you. Alert separately on the failure rate that feeds this whole workflow: a sustained non-2xx rate above roughly one percent for any single event type, or any occurrence at all of 401 where the baseline is zero, deserves a page. Trend the count of deliveries whose terminal state is failed rather than the raw count of failed attempts, because retries make the latter move for reasons that have nothing to do with a defect.

Finally, make the promotion path from evidence to fixture explicit. When a debugging session ends, the engineer should run one command that scrubs the delivery, writes it into the fixture directory with the bug’s name, and opens a change for review. If promotion requires hand-copying bytes out of a database, it will not happen, and the same failure will be debugged from scratch the next time it appears.

Frequently Asked Questions

Related

• Simulating webhook traffic spikes — reproducing load-induced timeouts at scale.
• Testing webhooks locally with ngrok and tunnels — exposing a local handler for replay.
• Recording and replaying webhook fixtures — promoting a reproduction into a permanent fixture.
• Inspecting and replaying webhook deliveries — the parent guide on capture and replay tooling.