Inspecting and Replaying Webhook Deliveries

Inspecting and replaying webhook deliveries is the operational backbone of Webhook Testing & Local Development: the ability to capture every incoming request exactly as it arrived, examine it after the fact, and re-send it through your handler on demand. Without this, a failed delivery is gone — you cannot see what the provider actually sent, and you cannot reproduce the bug. This guide is for engineers who already verify signatures and process events asynchronously and now need a durable record of raw deliveries plus safe tooling to replay them. The central discipline is preserving the exact raw bytes and headers so that both inspection and replay reflect reality, not a re-serialized approximation.

Deliveries are captured raw into an append-only store, inspected and diffed from the log, then replayed idempotently back through the handler.

Capture Patterns That Preserve Fidelity

Three capture approaches dominate, and they differ chiefly in how faithfully they preserve the original request. A request bin (a throwaway endpoint such as a self-hosted Webhook.site or RequestBin instance) is the fastest way to see what a provider sends during initial integration, but it lives outside your stack and should never hold production data. A middleware tap records every request inside your application before routing — the highest-fidelity option because it captures the exact raw byte stream at the socket boundary, headers included, prior to any framework parsing. A provider-side delivery log (Stripe, GitHub, and others expose recent deliveries with response codes) is authoritative for what the provider believes it sent, useful for reconciliation but limited by retention.

The non-negotiable rule across all three: store the raw body bytes verbatim. Re-serializing parsed JSON changes key order and whitespace, which breaks any later HMAC-SHA256 verification because the signature was computed over the original bytes. Persist the body as bytea/BLOB, alongside all headers, the receipt timestamp, the source IP, and your own correlation ID.

Inspection Workflows: Logs, Diffs, and Filtering

A useful delivery log is queryable, not a flat file. Index on event type, signature-validity, HTTP response code, and receipt time so you can answer “show every order.created.v1 that returned 5xx in the last hour” instantly. Inspection becomes powerful when you can diff two deliveries — comparing a working payload against a failing one frequently reveals a schema drift the provider never announced, which is exactly the failure your event schema design contracts are meant to prevent. Surface the computed-versus-received signature side by side so a mismatch is obvious, and decode and pretty-print the body for human reading without ever mutating the stored raw copy.

Operational Replay and CI/CD Integration

Replay re-injects a stored delivery into your handler, and it must be idempotent by construction: replaying an event that already produced a side effect must not duplicate it. Reuse the original event ID and your idempotency in webhooks store so a second run is a no-op. Replay has three production uses: recovering a real backlog after a deploy bug (replay the failed window), reproducing a customer issue locally (replay one delivery against a dev build through a tunnel), and seeding tests in CI (a corpus of captured real deliveries becomes a regression suite that proves a handler change still parses historical traffic). Always replay through the full verification path so the test exercises signing, not just business logic.

Failure Mode Analysis

Failure mode	Impact	Mitigation
Storing parsed JSON, not raw bytes	Signature verification fails on replay; debugging is misleading	Persist the exact raw body as binary alongside headers
Non-idempotent replay	Duplicate charges, emails, or state mutations	Reuse original event ID against an idempotency store before side effects
Capturing secrets in plaintext logs	Authorization headers and PII leak into the store	Redact sensitive headers; encrypt the store at rest
Unbounded delivery retention	Storage growth and compliance exposure	Apply TTL and field-level redaction aligned to retention policy
Replaying stale-timestamp events	Receiver rejects them as replay attacks	Re-sign with a fresh timestamp or bypass the window for trusted internal replay

Runnable Implementation Example

This Python capture-and-replay pair stores the raw delivery and re-injects it idempotently. The capture step deliberately reads request.get_data() (raw bytes) before any JSON parsing.

import hashlib
import hmac
import json
import time
import psycopg2

def capture(conn, raw_body: bytes, headers: dict, source_ip: str) -> str:
    """Persist the exact bytes and headers; never re-serialize the body."""
    correlation_id = hashlib.sha256(raw_body + str(time.time()).encode()).hexdigest()[:16]
    with conn.cursor() as cur:
        cur.execute(
            """INSERT INTO webhook_deliveries
                 (correlation_id, raw_body, headers, source_ip, received_at)
               VALUES (%s, %s, %s, %s, now())""",
            (correlation_id, psycopg2.Binary(raw_body),
             json.dumps(headers), source_ip),
        )
    conn.commit()
    return correlation_id


def replay(conn, correlation_id: str, target_url: str, secret: bytes,
           seen: set) -> dict:
    """Re-inject a stored delivery idempotently through the verifying handler."""
    with conn.cursor() as cur:
        cur.execute(
            "SELECT raw_body FROM webhook_deliveries WHERE correlation_id = %s",
            (correlation_id,),
        )
        row = cur.fetchone()
    if not row:
        return {"status": "not_found"}

    raw_body = bytes(row[0])
    event_id = json.loads(raw_body).get("id")
    if event_id in seen:                       # idempotency guard
        return {"status": "skipped", "reason": "already_processed"}

    # Re-sign with a fresh timestamp so the receiver's replay window accepts it.
    ts = str(int(time.time()))
    sig = hmac.new(secret, f"{ts}.".encode() + raw_body, hashlib.sha256).hexdigest()
    import requests
    resp = requests.post(
        target_url, data=raw_body,
        headers={"Content-Type": "application/json",
                 "X-Webhook-Signature": f"t={ts},v1={sig}",
                 "X-Replay-Source": "delivery-store"},
        timeout=10,
    )
    if resp.status_code < 300:
        seen.add(event_id)
        return {"status": "success"}
    return {"status": "failed", "code": resp.status_code}

Debugging Checklist

Confirm the stored body is byte-identical to what the provider sent (recompute and compare the signature).
Verify replay reuses the original event ID and is blocked by the idempotency store on a second run.
Check that Authorization and other sensitive headers are redacted or encrypted in the store.
Diff a failing delivery against a recent successful one to isolate schema drift.
Ensure replayed events carry a fresh timestamp (or trusted-replay flag) so the receiver does not reject them.
Validate that a CI replay corpus runs the full verification path, not just the business handler.

Debugging failed webhook deliveries — diagnosing and reproducing specific failures.
Load testing webhook endpoints — using captured traffic as realistic load.
Webhook contract testing — turning captured deliveries into contract fixtures.
Webhook Testing & Local Development — the broader testing discipline.