Idempotency keys vs deduplication windows for webhook consumers

Webhook providers deliver at-least-once, so a consumer will see the same event twice sooner or later. Two mechanisms suppress the duplicate effect: a durable idempotency key that records “this exact operation already ran,” and a time-bounded deduplication window that drops events seen recently within a sliding time horizon. This comparison builds on idempotency in webhooks and pairs with how to design idempotent webhook consumers, which walks through the key-extraction and atomic-store steps in depth. Here the goal is narrower: choose the right mechanism for a given workload.

The distinction matters because the two approaches fail differently. Keys give exact, permanent suppression at the cost of unbounded storage; windows give cheap, bounded storage at the cost of letting late duplicates through. Picking the wrong one shows up as either a runaway dedup table or a double-charged customer.

A key store suppresses duplicates forever but grows without bound; a window only remembers events inside its TTL and forgets the rest.

How an idempotency key works

An idempotency key is a stable identifier — usually supplied by the provider as X-Idempotency-Key, or derived as a hash of immutable payload fields — that you record the first time you process an event. Storage is typically a UNIQUE constraint in your database or a Redis SET key NX. The first insert wins and runs the side effect; every later insert with the same key collides and returns the cached result. Suppression is exact and permanent: even a duplicate that arrives a year later is caught, because the key is still on record.

The cost is storage that grows with event volume and an explicit decision about when, if ever, to prune it. Pruning a key reopens the window for a duplicate of that exact event.

How a deduplication window works

A deduplication window stores recently seen event identifiers with a TTL and rejects any identifier already present. In Redis this is a SET event_id "1" NX EX <seconds>; the entry self-expires, so the store stays bounded regardless of total volume. The window is sized to the provider’s retry horizon — if a provider retries for up to 72 hours, a 72-hour window catches every retry-driven duplicate.

The cost is that any duplicate arriving after the TTL expires slips through and re-runs the side effect. Windows trade exactness for bounded, self-cleaning storage.

Comparison

Dimension	Idempotency key	Deduplication window
Suppression guarantee	Exact and permanent	Only within the TTL horizon
Storage growth	Unbounded unless pruned	Bounded; entries self-expire
Late-duplicate handling	Always caught	Slips through after expiry
Implementation	`UNIQUE` constraint or `SET NX` + retention policy	`SET NX EX <ttl>`
Best fit	Money movement, account mutations, anything irreversible	High-volume idempotent-ish events, notifications, cache busts
Identifier source	Provider key or hash of immutable fields	Same, but only needs to be unique within the window
Operational risk	Table bloat, prune-too-early double-runs	TTL shorter than retry horizon lets duplicates through

When each one fits

Reach for a durable idempotency key when a duplicate side effect is expensive or irreversible: charging a card, transferring funds, provisioning a resource, or any operation where “ran twice” cannot be tolerated even months later. Pair the key with the atomic check-and-set described in how to design idempotent webhook consumers so concurrent retries cannot both win.

Reach for a deduplication window when events are high-volume and the cost of an occasional late duplicate is low — sending a notification, invalidating a cache, recomputing a derived value. The bounded, self-expiring store keeps Redis memory flat without a retention job.

Many production systems use both: a window absorbs the common case of rapid provider retries cheaply, while a durable key behind it guarantees exactness for the operations that truly cannot run twice.

A combined implementation

import hashlib
import json
import redis

r = redis.Redis(decode_responses=True)

def idempotency_id(payload: dict, header_key: str | None) -> str:
    # Prefer the provider-supplied key; fall back to a hash of immutable fields.
    if header_key:
        return header_key
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()

def process_with_window_and_key(payload: dict, header_key: str | None) -> str:
    eid = idempotency_id(payload, header_key)

    # Layer 1: cheap, bounded dedup window catches rapid provider retries.
    # TTL must be >= the provider's documented retry horizon.
    if not r.set(f"dedup:{eid}", "1", nx=True, ex=72 * 3600):
        return "duplicate-suppressed-by-window"

    # Layer 2: durable key guarantees exactness for irreversible effects.
    # A UNIQUE insert that fails means this operation already ran.
    if not r.set(f"idem:{eid}", "done", nx=True):  # no TTL: permanent
        return "duplicate-suppressed-by-key"

    run_side_effect(payload)  # charge, transfer, provision, etc.
    return "processed"

def run_side_effect(payload: dict) -> None:
    ...  # the irreversible work

The window short-circuits the storm of identical retries that arrive within seconds; the keyless-TTL idem: entry is the permanent backstop for the events that must never double-run.

Verification

A unit test should prove the window suppresses a second call and that the durable key survives window expiry.

import fakeredis

def test_window_suppresses_immediate_duplicate():
    fake = fakeredis.FakeStrictRedis(decode_responses=True)
    # First call processes; identical second call is suppressed.
    assert fake.set("dedup:abc", "1", nx=True, ex=10) is True
    assert fake.set("dedup:abc", "1", nx=True, ex=10) is None

def test_durable_key_outlives_window():
    fake = fakeredis.FakeStrictRedis(decode_responses=True)
    fake.set("idem:abc", "done", nx=True)          # permanent
    fake.set("dedup:abc", "1", nx=True, ex=1)       # window
    fake.delete("dedup:abc")                         # simulate TTL expiry
    # Window is gone, but the durable key still blocks a re-run.
    assert fake.set("idem:abc", "done", nx=True) is None

Failure modes and gotchas

TTL shorter than the retry horizon. A window sized at 1 hour against a provider that retries for 24 will let the 2-hour-late retry re-run the side effect. Always read the provider’s retry policy and size the window to its maximum, plus a margin.
Pruning durable keys too aggressively. Deleting old idempotency keys to reclaim space reopens the door for an exact duplicate of those events. If you must prune, archive instead and only delete keys older than any plausible duplicate.
Hashing mutable fields into the identifier. If the derived id includes a timestamp the provider rewrites on retry, every retry gets a fresh id and nothing is suppressed. Hash only fields that are stable across retries.
Race between window and key layers. With two stores, a crash between the window SET and the key SET can leave the window claimed but the effect un-run; a later retry is then suppressed by the window without ever executing. Make the durable key the source of truth and treat the window purely as an optimization, re-checking the key on any window miss.