Inspecting and Replaying Webhook Deliveries

Inspecting and replaying webhook deliveries is the operational backbone of Webhook Testing & Local Development: the ability to capture every incoming request exactly as it arrived, examine it after the fact, and re-send it through your handler on demand. Without this, a failed delivery is gone — you cannot see what the provider actually sent, and you cannot reproduce the bug. This guide is for engineers who already verify signatures and process events asynchronously and now need a durable record of raw deliveries plus safe tooling to replay them. The central discipline is preserving the exact raw bytes and headers so that both inspection and replay reflect reality, not a re-serialized approximation.

Capture Patterns That Preserve Fidelity

Three capture approaches dominate, and they differ chiefly in how faithfully they preserve the original request. A request bin (a throwaway endpoint such as a self-hosted Webhook.site or RequestBin instance) is the fastest way to see what a provider sends during initial integration, but it lives outside your stack and should never hold production data. A middleware tap records every request inside your application before routing — the highest-fidelity option because it captures the exact raw byte stream at the socket boundary, headers included, prior to any framework parsing. A provider-side delivery log (Stripe, GitHub, and others expose recent deliveries with response codes) is authoritative for what the provider believes it sent, useful for reconciliation but limited by retention.

The non-negotiable rule across all three: store the raw body bytes verbatim. Re-serializing parsed JSON changes key order and whitespace, which breaks any later HMAC-SHA256 verification because the signature was computed over the original bytes. Persist the body as bytea/BLOB, alongside all headers, the receipt timestamp, the source IP, and your own correlation ID.

Where the tap sits in the middleware chain determines whether any of this works. It must run before every component that could touch the body: JSON parsing, charset transcoding, gzip decoding, form decoding, and any “request normalisation” layer a framework installs by default. In practice that means the outermost middleware, reading the body once into memory and re-attaching it to the request object so downstream code still sees a readable stream. The symptom of getting this wrong is subtle and expensive: signatures verify correctly in production but fail on replay, because the captured bytes are the parsed-and-re-emitted version rather than what arrived. Prove the placement once with an assertion — recompute the signature from the stored bytes for a sample of live deliveries and alert if the match rate is not 100 percent — and you never have to wonder again.

Capture must also be unable to break ingestion. A synchronous INSERT on the request path adds its latency to every acknowledgment and hands the database the ability to take your endpoint down: when the capture table’s disk fills or an index bloats, deliveries start timing out and the provider begins retrying, which increases capture volume, which makes the problem worse. The safer shape is to write the capture record on a bounded background queue and treat a full queue as a reason to drop the capture, not the delivery. Emit a metric on every dropped capture so the gap is visible, and consider degrading gracefully — dropping the body but keeping the metadata row — before dropping the record entirely. Capture is diagnostic infrastructure; it should never be in a position to cause the incident it exists to explain.

Body size is the other operational limit worth deciding deliberately rather than discovering. Most providers send payloads under 32 KB, but bulk or batch events can reach several megabytes, and a handful of very large deliveries will dominate your storage bill and slow every query that selects the body column. Set an explicit capture ceiling — 256 KB is a reasonable default for most integrations — and when a body exceeds it, store the first N kilobytes, a truncation flag, the full byte length, and the SHA-256 of the complete body. The hash preserves the one property truncation would otherwise destroy: you can still prove whether two deliveries of the same event were byte-identical. For chunked transfer encoding, capture after reassembly but before any decoding, and record whether the body arrived compressed, because a Content-Encoding: gzip delivery signed over the compressed bytes will not verify against the decompressed ones.

Sampling deserves a sharper rule than “keep ten percent”. Sample the boring successes if volume demands it, but capture unconditionally on three classes: any delivery that produced a non-2xx response, any delivery whose signature failed to verify, and any event type you have received fewer than a few hundred times. That policy keeps the corpus interesting while letting a high-volume, well-understood event type be sampled at one percent. The trap of uniform sampling is that failures are rare by definition, so a ten-percent sample discards ninety percent of exactly the rows you will later wish you had.

Storage Layout and Retention Economics

A delivery store is one of the few tables in a webhook system whose growth is entirely predictable, which makes it one of the few you can size properly in advance. Multiply your sustained delivery rate by the average captured row size and you have the bill. At 20 deliveries per second with an 8 KB average body, the row plus headers plus indexes costs roughly 10 KB, which is about 17 GB per day and half a terabyte a month of primary database storage — enough that a naive “keep it all in Postgres forever” design becomes the largest table in the system within a quarter and starts affecting vacuum times and backup windows for everything else.

The fix is not shorter retention but tiered retention, because the value of a stored delivery drops far faster than its cost does. The first hours after a delivery are worth almost everything: that is when on-call is reading it. A week later, a customer report needs the body but rarely needs the free-text fields. A month later, a reconciliation job needs counts and codes, not payloads. Match the tiers to that curve.

Partition the hot table by receipt day. Dropping a partition is instant and produces no vacuum debt, whereas a DELETE of a day’s worth of rows from a single large table generates dead tuples proportional to your entire delivery volume and can stall autovacuum for hours. Daily partitions also make the tier transition trivial: the job that redacts day seven operates on exactly one partition, can be re-run idempotently, and can be verified by comparing row counts before and after. Move warm-tier bodies to object storage rather than keeping them in the primary database — they are read rarely, they compress well, and object storage lifecycle rules will handle the cold transition for you without another cron job to monitor.

Inspection Workflows: Logs, Diffs, and Filtering

A useful delivery log is queryable, not a flat file. Index on event type, signature-validity, HTTP response code, and receipt time so you can answer “show every order.created.v1 that returned 5xx in the last hour” instantly. Inspection becomes powerful when you can diff two deliveries — comparing a working payload against a failing one frequently reveals a schema drift the provider never announced, which is exactly the failure your event schema design contracts are meant to prevent. Surface the computed-versus-received signature side by side so a mismatch is obvious, and decode and pretty-print the body for human reading without ever mutating the stored raw copy.

Index for the three questions people actually ask, not for every column. A composite index on (event_type, received_at DESC) serves “show me recent deliveries of this type” and doubles as the range scan for a bulk replay. A partial index on received_at where response_code >= 400 is the one that matters most: failures are a small fraction of rows, so the partial index is a fraction of the size of a full one and answers the on-call question instantly even when the table holds hundreds of millions of rows. Add a unique index on the provider’s event ID if they supply one — it costs little and turns duplicate detection into a constraint rather than a query. Resist indexing the body; if you need to search inside payloads, extract the two or three fields you actually filter on into generated columns at capture time.

Diffing is the highest-value inspection tool and the easiest to get subtly wrong. A raw byte diff of two JSON bodies is dominated by noise — key order, whitespace, and IDs differ on every delivery — so the diff must run over a normalised view: keys sorted, volatile fields (event ID, timestamps, nonces) masked to a placeholder, numbers canonicalised. What survives that normalisation is the structural change you are hunting: a field that appeared, a field that vanished, an enum that gained a value, a number that became a string. Keep the normalisation rules in code next to the diff tool so everyone diffs the same way; two engineers using different masking rules will reach different conclusions about the same pair of deliveries.

The most useful diff is rarely the failing delivery against a hand-picked good one. It is the failing delivery against the most recent successful delivery of the same event type and schema version, chosen automatically. That comparison controls for everything except what changed, and it can be a single button in your inspection view. A close second is diffing the same event type across a version boundary, which is how you discover that the provider shipped a change they described as additive but which reordered an array or tightened a field’s type. When that happens, the finding belongs in your payload versioning discussion with the provider, not in a defensive patch to a single handler.

One rule protects everything above: the pretty-printed, decoded, syntax-highlighted view an engineer reads is a projection, generated on demand from the stored bytes and thrown away. The moment a tool writes its formatted version back to the store — or worse, stores only the formatted version because it was easier to read — every signature recomputation and every replay from that row becomes unreliable, and the corruption is invisible until someone tries to use it during an incident.

Operational Replay and CI/CD Integration

Replay re-injects a stored delivery into your handler, and it must be idempotent by construction: replaying an event that already produced a side effect must not duplicate it. Reuse the original event ID and your idempotency in webhooks store so a second run is a no-op. Replay has three production uses: recovering a real backlog after a deploy bug (replay the failed window), reproducing a customer issue locally (replay one delivery against a dev build through a tunnel), and seeding tests in CI (a corpus of captured real deliveries becomes a regression suite that proves a handler change still parses historical traffic). That corpus is also the raw material for webhook mocking and sandbox environments, where the same captured bytes are re-signed with a test key and served by a stand-in provider. Always replay through the full verification path so the test exercises signing, not just business logic.

Bulk Replay Safety Controls

Replaying one delivery is a debugging convenience. Replaying forty thousand is a production event, and it fails in ways a single replay never does. The three that actually bite are rate, ordering, and blast radius, and each needs an explicit control rather than an assumption.

Rate is the one people underestimate. A replay loop with no throttle will drive your handler at whatever rate the loop can issue requests — frequently ten to a hundred times normal traffic — and the resulting load lands on the same database, the same downstream APIs, and the same third-party integrations that live traffic depends on. Cap replay concurrency at a fraction of normal capacity, not a multiple: replaying at 20 percent of your steady-state throughput drains a 40,000-event backlog in a few hours while leaving headroom for real deliveries, and it keeps the replay from tripping the circuit breakers that protect your downstreams. Make the rate a runtime parameter you can lower without redeploying, because the right number is usually discovered ten minutes into the run.

Ordering matters whenever the events describe a mutable entity. Replaying order.updated events in insertion order is correct; replaying them in whatever order a parallel worker pool happens to finish them can leave an order in the state it held two hours before the window ended. If your handlers are genuinely commutative — pure upserts keyed on a version number that reject older versions — parallel replay is safe and you should say so explicitly in the tool. Otherwise replay serially per entity key, which is the same per-key ordering constraint live delivery lives under, and parallelise across keys instead of within them.

Blast radius is controlled by making every bulk replay a first-class, resumable job rather than a shell loop. It should have a dry-run mode that reports exactly how many deliveries match the selection and a sample of them; a persistent cursor so an interrupted run resumes instead of restarting from the beginning and re-processing everything; an automatic pause when the error rate over the last hundred replays exceeds a threshold; and an audit record naming the operator, the selection criteria, and the outcome. The automatic pause is the control that most often saves a night: it turns “we replayed 40,000 events into a broken handler” into “we replayed 300 and it stopped”.

Selection is part of safety too. Prefer selecting by the narrowest predicate that covers the incident — one event type, one tenant, one response code, one hour — over “everything that failed today”, and record the exact predicate in the audit row so the run can be reproduced or its scope proven afterwards. Where the failed work already landed in a dead-letter queue, drain from there rather than from the capture store: the queue already knows what remains outstanding, whereas the capture store contains successes and failures alike and it is easy to replay a delivery that was processed correctly the first time.

Redaction and Access Control for the Delivery Store

The capture store is a copy of every piece of customer data your provider has ever sent you, held in a table whose whole purpose is to be easy to query. Treat it with the access controls you would apply to the production database it mirrors, not the ones you apply to logs. Two specific controls do most of the work.

The first is redaction at write time for the fields that are never worth keeping. Authorization headers, cookies, API keys, and any bearer token in a header should be replaced with a fixed marker before the row is written — not filtered out at read time, because a read-time filter fails open the moment someone queries the table directly or restores a backup. Signature headers are the exception: keep them, because verification and diagnosis are impossible without them, and they are useless to an attacker without the secret. Payment card numbers and similar regulated fields, where a provider still sends them, should be masked in the body at capture, which means the stored body no longer verifies against the signature — an acceptable trade only for the specific event types that carry them, and one you should record explicitly so nobody later wonders why those rows fail a recomputation.

The second is access as an audited action. Reading a raw body is a privileged operation; make it one. Engineers should be able to see metadata and redacted bodies freely, and unsealing a raw body should require an explicit action that writes an audit row naming the operator and the correlation ID. This is not bureaucracy for its own sake: the delivery store is the most attractive single table in the system to an attacker who already has read access somewhere, and an audit trail is the difference between knowing what was exposed and guessing. Encrypt the store at rest with a key distinct from the primary application key, and make sure the replay tool is the only component with routine decrypt permission.

Finally, remember that replay is a write path. A tool that can re-inject arbitrary stored deliveries can also re-trigger every side effect your handlers perform, which makes it as sensitive as a database migration runner. Require an approval for bulk replays targeting production, keep the target URL on an allowlist so a mistyped host cannot send customer payloads to an arbitrary endpoint, and never let a replay tool authenticate with a credential broader than the endpoint it posts to.

Failure Mode Analysis

Runnable Implementation Example

This Python capture-and-replay pair stores the raw delivery and re-injects it idempotently. The capture step deliberately reads request.get_data() (raw bytes) before any JSON parsing.

import hashlib
import hmac
import json
import time
import psycopg2
import requests


def capture(conn, raw_body: bytes, headers: dict, source_ip: str) -> str:
    """Persist the exact bytes and headers; never re-serialize the body."""
    correlation_id = hashlib.sha256(raw_body + str(time.time()).encode()).hexdigest()[:16]
    with conn.cursor() as cur:
        cur.execute(
            """INSERT INTO webhook_deliveries
                 (correlation_id, raw_body, headers, source_ip, received_at)
               VALUES (%s, %s, %s, %s, now())""",
            (correlation_id, psycopg2.Binary(raw_body),
             json.dumps(headers), source_ip),
        )
    conn.commit()
    return correlation_id


def replay(conn, correlation_id: str, target_url: str, secret: bytes,
           seen: set) -> dict:
    """Re-inject a stored delivery idempotently through the verifying handler."""
    with conn.cursor() as cur:
        cur.execute(
            "SELECT raw_body FROM webhook_deliveries WHERE correlation_id = %s",
            (correlation_id,),
        )
        row = cur.fetchone()
    if not row:
        return {"status": "not_found"}

    raw_body = bytes(row[0])
    event_id = json.loads(raw_body).get("id")
    if event_id in seen:                       # idempotency guard
        return {"status": "skipped", "reason": "already_processed"}

    # Re-sign with a fresh timestamp so the receiver's replay window accepts it.
    ts = str(int(time.time()))
    sig = hmac.new(secret, f"{ts}.".encode() + raw_body, hashlib.sha256).hexdigest()
    resp = requests.post(
        target_url, data=raw_body,
        headers={"Content-Type": "application/json",
                 "X-Webhook-Signature": f"t={ts},v1={sig}",
                 "X-Replay-Source": "delivery-store"},
        timeout=10,
    )
    if resp.status_code < 300:
        seen.add(event_id)
        return {"status": "success"}
    return {"status": "failed", "code": resp.status_code}

Observability for the Capture and Replay Path

Capture and replay are infrastructure, and like all infrastructure they fail silently unless instrumented. Four signals are worth a dashboard and, in two cases, a page.

The first is capture coverage: the ratio of stored delivery rows to deliveries the endpoint actually acknowledged, computed per minute. It should sit at 1.0, or at your configured sampling ratio. Any sustained dip means the capture queue is shedding, and the consequence is invisible until an incident happens during the gap. Page on this if coverage for non-2xx deliveries — which should never be sampled away — drops below 1.0 at all.

The second is fidelity: for a small sample of stored deliveries, recompute the signature from the stored bytes and compare against the stored header. This is cheap to run as a scheduled job over a few hundred rows an hour, and it is the only check that catches a middleware reordering that silently starts capturing parsed bodies. A fidelity drop is a page, because every hour it goes unnoticed is an hour of unreplayable captures.

The third is tier-transition health: whether the redaction, archival, and purge jobs ran, how many rows each moved, and how long they took. A stalled redaction job is a compliance problem that will be found by an audit rather than by you; a purge job that suddenly moves ten times its usual row count means an upstream flood you have not noticed elsewhere.

The fourth is replay activity: replays started, deliveries replayed, replay error rate, and the age of the oldest in-flight bulk job. Tag every replayed request with a header such as X-Replay-Source and propagate it into your traces so replay traffic can be excluded from latency percentiles and business metrics. Without that tag, a large backlog recovery corrupts every dashboard it touches — a spike in “orders processed” that is really the same orders processed twice — and someone will make a decision on the bad number. Wire the correlation ID through as a trace attribute as well, so a delivery row, its handler span, and its replay attempts all join in your observability stack from a single identifier.

In CI, the corpus should run as a regular job rather than an occasional ritual: replay the stratified corpus against the built handler on every pull request, assert that each delivery produces the expected outcome, and fail the build on any parse error, unhandled exception, or newly rejected signature. Keep that job under a minute — engineers disable slow tests — and keep it hermetic by pointing it at a scratch database seeded per run, so the idempotency store starts empty and the assertions actually observe the side effects.

Debugging Checklist

• Confirm the stored body is byte-identical to what the provider sent (recompute and compare the signature).
• Verify replay reuses the original event ID and is blocked by the idempotency store on a second run.
• Check that Authorization and other sensitive headers are redacted or encrypted in the store.
• Diff a failing delivery against a recent successful one to isolate schema drift.
• Ensure replayed events carry a fresh timestamp (or trusted-replay flag) so the receiver does not reject them.
• Validate that a CI replay corpus runs the full verification path, not just the business handler.
• Confirm capture coverage is 1.0 for every delivery that returned a non-2xx response.
• Check that the redaction and purge jobs ran on schedule and moved the expected number of rows.
• Verify a bulk replay resumes from its cursor rather than restarting after an interruption.

Frequently Asked Questions

Related

• Debugging failed webhook deliveries — diagnosing and reproducing specific failures.
• Recording and replaying webhook fixtures — turning a captured corpus into re-signed test fixtures.
• Load testing webhook endpoints — using captured traffic as realistic load.
• Webhook contract testing — checking captured deliveries against the published schema.
• Webhook Testing & Local Development — the broader testing discipline.