Replay Attack Prevention: Webhook Deduplication & Idempotency Patterns

Threat Model & Architectural Positioning

Replay attacks exploit intercepted payloads by retransmitting them to consumer endpoints, triggering duplicate state mutations, double-charging, or unauthorized resource provisioning — one of the sharpest edges in Webhook Security, Signing & Validation. Within that broader framework, cryptographic signatures alone cannot prevent retransmission. Signatures verify origin and integrity but remain valid indefinitely unless paired with temporal or stateful constraints. Effective mitigation requires deterministic validation layers that operate independently of payload content, enforce strict execution boundaries, and guarantee exactly-once processing semantics.

Attacker Capabilities Worth Modelling

Designing this layer well means being specific about who can capture a delivery in the first place, because the answer determines which control is load-bearing. Four capability levels cover nearly every real incident, and they demand different responses.

The weakest and most common is the passive log reader: someone with access to an aggregated log store, an APM trace, or an error-reporting service where a well-meaning engineer logged a full request including headers and body. They cannot intercept live traffic, but they can replay anything the retention window still holds — often 30 days. Against this attacker the timestamp window is decisive, because the captured payload is almost always long stale by the time anyone finds it.

Next is the endpoint neighbour: a tenant, a contractor, or a compromised internal service that legitimately receives some deliveries and re-sends them, either to a different endpoint or back to the same one. Timestamps do not help if they act quickly, so the nonce gate carries the weight. Above that sits the network-position attacker who can observe or delay traffic — a hostile proxy, a compromised egress gateway, a rogue mobile network. This one can also withhold a delivery and release it later, which is the case that makes an unsigned timestamp header worthless: they simply rewrite it to the current time before forwarding.

The strongest is the producer-side compromise, where an attacker can mint new, correctly signed deliveries. No replay control helps here, and pretending otherwise is the most common analytical error in this area. Signing key compromise is addressed by rotation and by storing webhook secrets in a secrets manager, not by deduplication. Being explicit about this boundary keeps the replay layer from accumulating expensive features that solve a threat it structurally cannot address.

What Each Control Blocks and What It Misses

Three controls are routinely described as interchangeable and are not: the timestamp window, the nonce cache, and an idempotent handler. Each blocks a different subset of duplicates, and the useful design question is which combination leaves no gap you care about.

Read down the last row and the layering rationale becomes obvious. The timestamp check needs nothing but a clock, so it keeps working through a total cache outage; the nonce gate goes blind at precisely the moment an attacker who is watching your status page would choose to act. That asymmetry is the argument for ordering the gates as signature, then timestamp, then nonce — the cheapest and most available checks reject the most traffic, and forged or stale requests never reach the cache at all. It is also the argument against the common shortcut of skipping timestamps once a nonce cache exists: doing so makes an entire class of attack contingent on Redis being up.

The idempotent handler occupies a different position again. It absorbs duplicates rather than rejecting them, which means the work has already been attempted by the time it helps. That is perfectly adequate for a pure database upsert and inadequate the moment the handler sends an email, charges a card, or calls a partner API, because those side effects happen outside the transaction that the uniqueness constraint protects. Design the handler to be idempotent regardless — it is the backstop when the other two gates are misconfigured — but do not treat it as a substitute for rejecting the duplicate earlier. The consumer-side patterns for this are worked through in how to design idempotent webhook consumers.

Core Deduplication Mechanisms

The foundational control couples payload verification with unique request identifiers. While HMAC Signature Verification guarantees data integrity and origin authenticity, it lacks temporal awareness. Production systems must implement an atomic deduplication layer using distributed caches to track processed nonces or idempotency keys, enforcing single-use constraints before business logic execution. The canonical implementation of this gate is covered in nonce-based replay protection with Redis. The deduplication layer must support high-throughput atomic writes, sliding expiration, and fallback persistence to relational databases for auditability.

Temporal Validation & Clock Synchronization

Time-bound validation windows introduce operational resilience against captured payloads. Implementing Preventing webhook replay attacks with timestamps establishes a sliding acceptance threshold. Endpoints must reject requests exceeding a configurable tolerance window while maintaining strict NTP synchronization across producer and consumer infrastructure to prevent false rejections from clock drift. Tolerance windows typically range from ±30 seconds to ±5 minutes, depending on network topology and delivery guarantees; the trade-off between replay exposure and false rejections is quantified in choosing a timestamp tolerance window.

Sizing the Window Against Measured Drift

The tolerance window is the one number on this page that should never be copied from another system, because it is entirely determined by two quantities you can measure: worst-case clock offset between producer and consumer, and worst-case delivery transit time. Set it as the sum of those two with a safety multiplier, and nothing else.

Start with the clock. A host running chrony against a nearby pool typically holds an offset under 5 milliseconds and rarely exceeds 50 milliseconds even after a network blip. Virtual machines that were suspended and resumed, and containers on oversubscribed hosts, are the outliers — offsets of several seconds appear after a live migration. Take the 99.9th percentile of your own measured offset rather than the median; if you are not exporting that metric today, that is the first thing to add, because you cannot size this window without it.

Then the transit time. This is not your endpoint’s response latency. It is the age of the timestamp when the request arrives, which includes the producer’s own queueing delay. A provider that batches or retries can present a timestamp several minutes old on a perfectly legitimate delivery, and providers rarely document this. Measure it: log the difference between the signed timestamp and the receipt time for a week of real traffic and look at the tail, not the mean.

The cost of getting this wrong is asymmetric and that should shape the default. Too narrow, and you reject legitimate deliveries — a visible, immediate, self-inflicted outage that pages someone. Too wide, and you extend the interval in which a captured payload is replayable, which is invisible until it is exploited. Because the narrow failure is loud and the wide failure is silent, teams reliably over-correct toward wide windows, so make the decision explicitly and revisit it with data rather than drifting upward one incident at a time. A useful discipline is to treat any widening as a temporary mitigation with an expiry date attached, and to require the measured drift number in the ticket that requests it.

One derived rule keeps the two gates consistent: the nonce TTL must be at least as long as the tolerance window, and there is no benefit to making it longer. Shorter, and a replay that arrives after the key expires but before the timestamp goes stale passes both gates — the exact seam an attacker would probe. Longer, and you are paying memory to remember identifiers that the timestamp gate would reject on its own. Derive both from a single constant so they cannot drift apart during a config change.

Token Lifecycle & Stateful Binding

For stateful or session-aware integrations, ephemeral credentials provide an additional replay barrier. When integrated with JWT-Based Webhook Auth, the jti (JWT ID) claim enforces strict single-use validation, while short expiration policies automatically invalidate intercepted tokens. This approach shifts replay risk from persistent storage to cryptographic expiration, reducing cache footprint and simplifying garbage collection of consumed identifiers. Whichever identifier you use — a provider nonce, an idempotency key, or a jti — it moves through exactly three states, and the only dangerous one is the return to unseen.

Implementation Blueprint

The following production-grade Python implementation demonstrates the required validation sequence: signature verification → timestamp validation → idempotency check → payload processing → nonce persistence. It utilizes Redis for atomic deduplication with SET ... NX and configurable TTL.

import time
import hashlib
import hmac
import os
import redis
from typing import Optional
from fastapi import FastAPI, Request, HTTPException, status

app = FastAPI()

SHARED_SECRET = os.environb.get(b"WEBHOOK_SIGNING_SECRET", b"")
REDIS_CLIENT = redis.Redis(host="localhost", port=6379, decode_responses=True)
TIMESTAMP_TOLERANCE_SEC = 300  # 5 minutes
IDEMPOTENCY_TTL_SEC = 900       # 15 minutes

def verify_hmac(payload: bytes, signature: str) -> bool:
    expected = hmac.new(SHARED_SECRET, payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)

def validate_timestamp(timestamp_header: Optional[str]) -> bool:
    if not timestamp_header:
        return False
    try:
        request_ts = int(timestamp_header)
        current_ts = int(time.time())
        return abs(current_ts - request_ts) <= TIMESTAMP_TOLERANCE_SEC
    except ValueError:
        return False

@app.post("/webhooks/events")
async def handle_webhook(request: Request):
    # 1. Extract headers and payload
    signature = request.headers.get("X-Webhook-Signature")
    timestamp = request.headers.get("X-Webhook-Timestamp")
    idempotency_key = request.headers.get("X-Idempotency-Key")
    payload_bytes = await request.body()

    # 2. Verify cryptographic signature
    if not signature or not verify_hmac(payload_bytes, signature):
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid signature"
        )

    # 3. Validate temporal window
    if not validate_timestamp(timestamp):
        raise HTTPException(
            status_code=status.HTTP_400_BAD_REQUEST, detail="Expired timestamp"
        )

    # 4. Atomic deduplication check
    if not idempotency_key:
        raise HTTPException(
            status_code=status.HTTP_400_BAD_REQUEST, detail="Missing idempotency key"
        )

    # SET ... NX returns True if key was set (new), False if it already existed
    is_new = REDIS_CLIENT.set(
        idempotency_key, "1", nx=True, ex=IDEMPOTENCY_TTL_SEC
    )
    if not is_new:
        # Idempotent response: return 200 OK without reprocessing
        return {"status": "already_processed", "key": idempotency_key}

    # 5. Process business logic (exactly-once execution guaranteed)
    try:
        process_event(payload_bytes)
        return {"status": "accepted", "key": idempotency_key}
    except Exception as e:
        # Rollback nonce on failure to allow retry
        REDIS_CLIENT.delete(idempotency_key)
        raise HTTPException(
            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=str(e)
        )

def process_event(payload: bytes) -> None:
    # Business logic implementation
    pass

Two details in that ordering are deliberate and frequently got wrong. The signature check runs before anything touches Redis, so a flood of forged requests costs one HMAC computation each and never reaches the cache — an attacker cannot use unauthenticated traffic to exhaust your dedup store or to enumerate which identifiers you have already seen. And the claim is a single SET ... NX, not an EXISTS followed by a SET; the two-call version has a window of a millisecond or so in which two concurrent deliveries of the same event both observe “not seen” and both proceed, which is precisely the duplicate you built the layer to prevent. Under a producer that fans out retries in parallel, that race fires far more often than its width suggests, because retries of the same event are correlated in time by construction.

The rollback on failure deserves scrutiny rather than acceptance. Deleting the key when processing throws makes the delivery retryable, which is usually what you want — but it also means a handler that fails after a side effect has already landed will happily run that side effect again on retry. Delete the nonce only for errors you know occurred before any external effect, and for everything else leave the key in place and route the event to a dead-letter queue for human triage. Distinguishing those two cases explicitly, rather than catching a bare exception, is the difference between a retry system and a double-charge incident.

Degraded Mode: Fail Open or Fail Closed

Every deduplication design eventually meets the question of what to do when the cache is unreachable, and the answer is not global. Fail closed and a Redis incident becomes a full delivery outage, with the producer’s retry queue backing up and — depending on the provider — events being dropped once their retry budget is exhausted. Fail open and the replay gate silently disappears for the duration. Both are defensible; picking one for the whole system is not.

Encoding the policy per event class makes the behaviour reviewable and testable instead of being an argument at three in the morning:

from enum import Enum

class DegradedPolicy(Enum):
    FAIL_OPEN = "fail_open"       # duplicate is harmless; keep accepting
    QUARANTINE = "quarantine"     # accept, park, reconcile later
    FAIL_CLOSED = "fail_closed"   # reject; a duplicate is worse than a delay

# The catalogue is the source of truth, reviewed like any other schema change.
DEGRADED_POLICY = {
    "invoice.updated": DegradedPolicy.FAIL_OPEN,
    "subscription.renewed": DegradedPolicy.QUARANTINE,
    "payout.created": DegradedPolicy.FAIL_CLOSED,
    "refund.issued": DegradedPolicy.FAIL_CLOSED,
}
DEFAULT_POLICY = DegradedPolicy.FAIL_CLOSED  # unknown events fail safe

def on_dedup_unavailable(event_type: str, payload: bytes) -> str:
    """Called only when the nonce store raised; the signature and timestamp
    gates have already passed, so this is a legitimate, fresh delivery."""
    policy = DEGRADED_POLICY.get(event_type, DEFAULT_POLICY)
    if policy is DegradedPolicy.FAIL_OPEN:
        process_event(payload)
        return "processed_without_dedup"
    if policy is DegradedPolicy.QUARANTINE:
        park_for_reconciliation(event_type, payload)
        return "quarantined"
    raise HTTPException(
        status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
        detail="deduplication unavailable; retry later",
    )

def park_for_reconciliation(event_type: str, payload: bytes) -> None:
    # Append to a durable store that a reconciliation job drains once the
    # dedup gate is healthy again. Never silently discard.
    pass

Defaulting unknown event types to fail-closed is the important line. New event types appear faster than anyone updates a policy table, and the failure you want from an unclassified event is a retryable rejection rather than an unguarded write. Pair the policy with a bounded degraded period: if the cache has been unavailable for longer than the producer’s total retry budget, fail-closed stops being a delay and becomes data loss, so the incident response has to escalate to restoring the cache rather than waiting it out.

Keyspace Design and Memory Budgeting

The nonce store is a capacity-planned component, not a convenience. Under-provision it and eviction quietly reopens the replay window; over-provision it and you have paid for a Redis instance three times larger than necessary. The arithmetic is simple enough to do on the back of an envelope before you deploy.

Budget memory from peak sustained throughput rather than average. A key like the one above is roughly 40 bytes, the value is a single byte, and Redis overhead for a small string with a TTL runs about 70 to 100 bytes once the expiry metadata and hash-table entry are counted — call it 150 bytes per nonce as a planning figure. At 2,000 deliveries per second with a 300-second TTL you hold 600,000 keys, or roughly 90 MB. At 20,000 per second you hold 6 million keys and about 900 MB, which is the point at which a shared cache instance stops being appropriate. Add 30% headroom for a retry storm, because a producer working through a backlog can briefly deliver at several times its steady rate.

Two configuration choices follow directly from that number. Run the dedup keyspace on an instance with maxmemory-policy noeviction so a memory ceiling produces loud write errors that your degraded-mode path handles, rather than an LRU eviction that silently deletes nonces still inside their window — the failure that turns a capacity problem into a security problem with no log line. And keep it off the instance holding your session or fragment cache, where a traffic spike in an unrelated feature can evict your nonces. A dedicated instance is not overkill here; it is the only way the memory budget you calculated remains meaningful.

Failure Mode Analysis & Troubleshooting

Distributed deduplication introduces specific failure vectors that require explicit mitigation strategies and operational runbooks.

Explicit Troubleshooting Runbook

1. Nonce Collision Rate > 0.1%: Indicates key generation weakness or cache eviction misalignment. Verify UUIDv4/v7 generation entropy. Adjust volatile-lru to noeviction if memory permits, or increase cluster capacity.
2. Timestamp Rejection Spike > 3σ: Correlate with network latency or NTP desync. Enable tcpdump on webhook ingress to measure producer-to-consumer transit time. Adjust TIMESTAMP_TOLERANCE_SEC dynamically via feature flag.
3. Deduplication Latency > 50ms p99: Redis pipeline contention or network partition. Implement connection pooling, enable pipeline() for batch nonce checks, and route traffic via consistent hashing to dedicated cache shards.

Operational Workflows & Monitoring

Deployment follows a phased validation pipeline to ensure zero-downtime integration:

1. Static Analysis: Lint validation logic for cryptographic timing attacks and race conditions.
2. Synthetic Replay Injection: Generate duplicate payloads with identical X-Idempotency-Key and expired timestamps to verify rejection paths.
3. Canary Deployment: Route 5% shadow traffic through the deduplication layer while comparing processing outcomes against baseline consumers.
4. Full Rollout: Enable real-time deduplication metrics and activate automated scaling policies.

Two things make or break that rollout in practice. First, the canary comparison has to run in shadow rather than in-line: send the duplicate through the new gate but let the existing consumer keep processing, and compare decisions offline. An in-line canary that rejects a delivery the old path would have accepted turns a validation exercise into an incident. Second, the synthetic replay injection needs to include the awkward cases, not just an exact duplicate — the same event with a fresh trace identifier, the same event arriving one second before and one second after the window boundary, and two copies arriving concurrently on different consumer instances. Each of those exercises a different line of the implementation, and the concurrent pair is the only test that catches a non-atomic claim.

What the Metrics Actually Tell You

The duplicate-rejection counter is the most misread number in this layer. A count of zero is not a clean bill of health; it is far more likely to mean the gate is not working — an unstable identifier, a keyspace prefix that changed, a client pointed at a different cache. Healthy production systems reject a steady trickle of duplicates because producers legitimately retry after a lost response, so treat a drop to zero as an alertable regression rather than a success. Set the alert on the absence of rejections over a multi-hour window, sized against your normal baseline.

Similarly, a spike in timestamp rejections almost never means an attack. In order of likelihood it means an NTP problem on one consumer node, a producer whose queue backed up and is now flushing hours-old events, or someone shipping a change to the tolerance constant. Include the signed timestamp age distribution in the alert payload — not just the rejection count — because the shape of that distribution distinguishes all three instantly: clock drift shifts the whole distribution, a flushing backlog stretches the upper tail, and a config change produces a cliff at the new boundary.

Track deduplication latency as a share of total handler time rather than in absolute milliseconds. A p99 of 40 ms is fine in a handler that takes 400 ms and is a serious problem in one that takes 15 ms, because in the second case the gate has become the dominant cost and a cache hiccup will dominate your delivery SLO. When the ratio exceeds about 20%, the fix is usually connection pooling or moving the cache into the same availability zone, not a bigger instance.

Monitoring Thresholds

• Nonce collision rate: Alert if > 0.1% over 5-minute window
• Timestamp rejection ratio: Alert if spike exceeds 3σ baseline deviation
• Cache hit ratio: Maintain ≥ 95%; trigger scale-up if < 90%
• Deduplication latency: SRE page if p99 > 50ms

Pre-Rollout Verification Checklist

• The nonce TTL and the timestamp tolerance window are derived from one shared constant.
• The dedup claim is a single atomic SET ... NX PX, never EXISTS followed by SET.
• Signature and timestamp checks run before any Redis write, so forged traffic costs nothing.
• A synthetic duplicate delivery is asserted in CI to return the idempotent response, not a second execution.
• Redis unavailability has an explicit, documented fail-closed or fail-open decision per event class.
• chronyd/ntpd offset is alerted on at ±50ms across every consumer node.

Incident Response Protocol

1. Quarantine: Immediately isolate affected endpoints behind API gateway WAF rules.
2. Rotate: Invalidate compromised signing keys and issue new HMAC secrets via secure key management.
3. Audit: Parse consumer logs for duplicate executions using idempotency_key traces.
4. Recover: Replay legitimate missed events from dead-letter queues with regenerated nonces and updated timestamps.

The recovery step hides a subtlety worth stating plainly: replaying from a dead-letter queue means deliberately re-sending events that your own defenses are built to reject. Regenerating nonces and timestamps for a legitimate replay is correct, but the mechanism that does it is, by construction, a bypass of the replay gate. Restrict it to an authenticated internal path, require an operator identity on every invocation, and log the original event identifier alongside the regenerated one so the audit trail survives. Teams that instead widen the tolerance window to let a bulk replay through end up leaving it widened, which is how a five-minute window quietly becomes an hour.

Frequently Asked Questions

Related

• Preventing Webhook Replay Attacks with Timestamps — the temporal validation gate in depth, with fail-closed middleware.
• Nonce-Based Replay Protection with Redis — atomic single-use enforcement using SET ... NX.
• Choosing a Timestamp Tolerance Window — sizing the window against measured drift and transit latency.
• HMAC Signature Verification — the integrity layer replay defenses build on top of.
• Webhook Security, Signing & Validation — the full security discipline this control belongs to.