Webhook Architecture & Event-Driven Integration

Webhook Delivery Queue Architecture: Brokers, Retry Tiers and Worker Topology

Every durable webhook sender in Resilient Delivery & Retry Strategies eventually resolves into the same shape — a queue between the thing that produced an event and the thing that makes an HTTP request — and almost every delivery incident traces back to a decision made about that queue rather than about the HTTP call itself. The dispatcher is easy: sign the body, POST it, read the status code. The queue is where the hard properties live. How long may a worker hold a message before the broker assumes it died? Where does a message wait out a 40-second backoff without pinning a thread? Does a single tenant’s 200,000-event backlog sit in front of everyone else’s password-reset webhooks?

This guide covers the topology, not the failure sink. The dead-letter queue architecture pages deal with what happens after a delivery is given up on; here the subject is the live path — the ready queue, the delay tiers that implement backoff, the leases that make at-least-once delivery honest, and the partitioning decisions that bound how much one noisy publisher can hurt everyone else.

Where the queue sits in a webhook dispatcher

A production dispatcher has at least three logical queues, even if a small deployment collapses them onto one broker. The ready queue holds envelopes eligible for an attempt right now. The delay queue (or ladder of delay queues) holds envelopes waiting out a backoff interval. The dead-letter sink holds envelopes whose retry budget is spent. Workers only ever read from the ready queue; everything else is a scheduler feeding it.

That separation matters because the three queues have completely different shapes. The ready queue should be shallow — if it is deep, you are under-provisioned. The delay queue is supposed to be deep; depth there is just the integral of your backoff schedule and says nothing about health. The dead-letter sink should be near-empty and any growth is an incident. Alerting on “queue depth” without distinguishing the three produces pages that mean nothing, which is why the metric that actually matters is the age of the oldest ready message.

Webhook dispatch queue topology Events flow from the producer into a ready queue, are leased by the dispatch pool and delivered to the endpoint; failed attempts move to a delay queue that re-queues them after backoff, and exhausted attempts land in the dead-letter sink. Producer domain event Ready queue eligible now Dispatch pool leased workers Endpoint tenant HTTPS enqueue lease 2xx Delay queue backoff tiers 5xx, timeout, 429 delay elapsed Dead-letter manual triage budget exhausted
Workers only ever read the ready queue; the delay tiers and the dead-letter sink are fed by the scheduler, which is why their depths must be alerted on separately.

The arrow that people forget is the one going back from the delay queue into the ready queue. That path is a producer, and it competes with the real producer for ready-queue capacity. During a large provider outage the retry path can produce far more messages per second than the business does, which is the mechanism behind most retry storms. Bounding it is the job of exponential backoff algorithms and, when an endpoint is comprehensively down, of circuit breaker patterns that stop generating attempts at all.

Leases, visibility timeouts and honest at-least-once semantics

Queues do not hand a message to a worker; they lease it. The broker makes the message invisible for a configured window — SQS calls it the visibility timeout, RabbitMQ implements it as an unacknowledged delivery on a channel, Redis Streams as a pending entry with an idle time. If the worker acknowledges within the window, the message is deleted. If it crashes, the lease expires and the message becomes visible to someone else. That is the entire mechanism behind at-least-once delivery, and every duplicate webhook your consumers ever see comes out of it.

The failure mode is mundane and extremely common: the lease is shorter than the worst-case HTTP attempt. A worker POSTs to an endpoint that takes 45 seconds to respond while the visibility timeout is 30 seconds. At second 30 the broker re-leases the message, a second worker POSTs the same event, and the tenant receives the webhook twice — while both workers eventually get a 200 and both try to delete a message that has already moved on. The rule is simple: the lease window must exceed connect timeout plus read timeout plus scheduling slack, and your HTTP client must actually enforce those timeouts. If the endpoint’s response time is unpredictable, cap it explicitly as described in handling slow webhook consumers.

Envelope lifecycle state machine An envelope moves from queued to in flight when leased, returns to queued if the lease expires, moves to delivered on a 2xx, to a scheduled retry on failure, and to dead-lettered when the attempt cap is reached. Queued In flight lease held Delivered Scheduled waiting out backoff Dead-lettered receive lease expired 2xx then ack 5xx or timeout backoff elapsed cap hit
Only two edges leave "in flight" by design; the third — lease expiry — is the one that manufactures duplicates when the timeout is set below the real request budget.

Because that edge exists, the queue cannot promise exactly-once delivery no matter what the broker’s marketing says, and the correct response is to make the consumer side tolerate repeats rather than to chase a stronger guarantee. That trade-off is laid out in at-least-once vs exactly-once delivery trade-offs, and the practical consumer contract in how to design idempotent webhook consumers. Send a stable Idempotency-Key header derived from the envelope ID — never regenerated per attempt — and duplicates become a non-event.

A second, quieter consequence of leasing: the in-flight count is a real capacity limit. SQS allows 120,000 in-flight messages per standard queue and 20,000 for FIFO; RabbitMQ bounds it with per-channel prefetch. If your workers hold long leases on a large pool, you can hit that ceiling and the broker will simply stop delivering messages while the queue still shows depth — a genuinely confusing outage where nothing is broken and nothing is moving.

Three topologies for the retry path

The interesting design choice is not “which broker” but “how many queues, and what distinguishes them”. Three patterns cover nearly all production webhook senders.

Pattern A — single work queue with in-process retry. One queue, one worker pool; failed attempts sleep inside the worker and retry in place while holding the lease. Attractive because it is fifty lines of code, and it is fine below roughly ten deliveries per second with sub-second endpoints. It breaks the moment backoff exceeds the lease window: a worker sleeping 60 seconds either loses its lease (duplicate) or must heartbeat the lease (complexity), and it burns a concurrency slot doing nothing. Every worker asleep is a worker not delivering.

Pattern B — a retry ladder of delay queues. The ready queue holds first attempts. On failure, the envelope is written to a delay tier — retry.10s, retry.1m, retry.15m, retry.2h — and a scheduler promotes it back to ready when its due time passes. Workers never sleep. Depth in each tier is directly readable as “how many deliveries are currently in backoff at this stage”, which is the single most useful diagnostic a webhook platform can have. The cost is more moving parts and a scheduler you must keep running.

Pattern C — partitioned or per-tenant queues. Ready traffic is split by tenant, either into physically separate queues or into partitions of one queue keyed by tenant ID. This is the only pattern that gives real isolation, and it is the only one where one customer’s outage cannot consume the whole worker pool. It is also the one that generates operational sprawl; the shard-explosion problem and the alternatives are worked through in partitioning webhook queues by tenant.

Queue topology comparison matrix Five criteria compared across a single shared queue, a tiered retry ladder and a queue per tenant, showing isolation, object count, backoff fidelity, operational overhead and ordering scope. Criterion Single shared queue Tiered retry queues Queue per tenant Tenant isolation none none strong Broker objects one four to six one per tenant Backoff fidelity worker sleeps per-tier delay per-tier delay Ops overhead minimal low high Ordering scope best effort best effort per tenant
The ladder buys observable backoff for almost no operational cost; only per-tenant separation buys isolation, and it is the expensive column for a reason.

Most platforms end up at B plus a narrow slice of C: a shared ladder for the great majority of tenants, and dedicated queues for the handful of accounts big enough to hurt the rest. Choosing A permanently is a decision to cap your throughput at whatever a sleeping worker pool can sustain.

Scheduling backoff without pinning a worker

The delay tier is the piece brokers implement least consistently, and it drives more broker selection decisions than anything else. SQS has per-message DelaySeconds but caps it at 900 seconds, so a 2-hour backoff needs either a chain of delay queues or an external scheduler. RabbitMQ has no native per-message delay; the idiom is a TTL queue with a dead-letter exchange pointing back at the work queue, one queue per delay value, or the delayed-message-exchange plugin. Kafka has nothing at all — the community pattern is a set of retry topics with a consumer that pauses partitions until the record’s due time. Redis Streams likewise has no delay, but a sorted set keyed by due timestamp is trivial and exact to the millisecond. Those differences are compared head to head in choosing a message broker for webhook delivery.

The implementation below is the Redis variant, because it shows every moving part explicitly: a ready list, a scheduled sorted set, an in-flight sorted set that doubles as the lease table, and a reaper that returns abandoned leases. It runs as written against Redis 7 with redis-py 5.0 and httpx 0.27.

"""Webhook dispatch worker with a Redis-backed retry ladder.

Keys
  wh:ready      LIST  envelope ids eligible for an attempt now
  wh:scheduled  ZSET  member=envelope id, score=unix ts when it becomes due
  wh:inflight   ZSET  member=envelope id, score=unix ts when the lease expires
  wh:dead       LIST  envelope ids whose retry budget is spent
  wh:body       HASH  envelope id -> JSON envelope
"""
import json
import os
import random
import signal
import time
import uuid

import httpx
import redis

R = redis.Redis.from_url(os.environ.get("REDIS_URL", "redis://localhost:6379/0"))

READY, SCHEDULED, INFLIGHT, DEAD, BODY = (
    "wh:ready", "wh:scheduled", "wh:inflight", "wh:dead", "wh:body",
)

CONNECT_TIMEOUT = 3.0
READ_TIMEOUT = 10.0
LEASE_SECONDS = 45          # > connect + read + scheduling slack
MAX_ATTEMPTS = 8
BASE_DELAY = 2.0
MAX_DELAY = 7200.0
RETRYABLE_STATUS = {408, 425, 429, 500, 502, 503, 504}


def enqueue(tenant: str, endpoint: str, event: dict) -> str:
    """Called by the producer (ideally from an outbox, not a request handler)."""
    envelope_id = uuid.uuid4().hex
    envelope = {
        "id": envelope_id,
        "tenant": tenant,
        "endpoint": endpoint,
        "event": event,
        "attempt": 0,
        "enqueued_at": time.time(),
    }
    pipe = R.pipeline()
    pipe.hset(BODY, envelope_id, json.dumps(envelope))
    pipe.lpush(READY, envelope_id)
    pipe.execute()
    return envelope_id


def promote_due(batch: int = 500) -> int:
    """Scheduler tick: move envelopes whose backoff has elapsed onto ready."""
    now = time.time()
    due = R.zrangebyscore(SCHEDULED, "-inf", now, start=0, num=batch)
    if not due:
        return 0
    pipe = R.pipeline()
    for envelope_id in due:
        pipe.zrem(SCHEDULED, envelope_id)
        pipe.lpush(READY, envelope_id)
    pipe.execute()
    return len(due)


def reap_expired_leases(batch: int = 500) -> int:
    """At-least-once safety net: a worker that died mid-attempt loses its lease."""
    now = time.time()
    expired = R.zrangebyscore(INFLIGHT, "-inf", now, start=0, num=batch)
    if not expired:
        return 0
    pipe = R.pipeline()
    for envelope_id in expired:
        pipe.zrem(INFLIGHT, envelope_id)
        pipe.lpush(READY, envelope_id)
    pipe.execute()
    return len(expired)


def lease(block_seconds: int = 5) -> dict | None:
    item = R.brpop(READY, timeout=block_seconds)
    if item is None:
        return None
    envelope_id = item[1].decode()
    R.zadd(INFLIGHT, {envelope_id: time.time() + LEASE_SECONDS})
    raw = R.hget(BODY, envelope_id)
    if raw is None:                       # body reaped by retention; drop the lease
        R.zrem(INFLIGHT, envelope_id)
        return None
    return json.loads(raw)


def backoff_seconds(attempt: int) -> float:
    """Full-jitter exponential backoff, capped at MAX_DELAY."""
    ceiling = min(MAX_DELAY, BASE_DELAY * (2 ** attempt))
    return random.uniform(BASE_DELAY, ceiling)


def complete(envelope: dict) -> None:
    pipe = R.pipeline()
    pipe.zrem(INFLIGHT, envelope["id"])
    pipe.hdel(BODY, envelope["id"])
    pipe.execute()


def reschedule(envelope: dict, reason: str) -> str:
    envelope["attempt"] += 1
    envelope["last_error"] = reason
    pipe = R.pipeline()
    pipe.zrem(INFLIGHT, envelope["id"])
    pipe.hset(BODY, envelope["id"], json.dumps(envelope))
    if envelope["attempt"] >= MAX_ATTEMPTS:
        pipe.lpush(DEAD, envelope["id"])
        outcome = "dead_lettered"
    else:
        pipe.zadd(SCHEDULED, {envelope["id"]: time.time() + backoff_seconds(envelope["attempt"])})
        outcome = "rescheduled"
    pipe.execute()
    return outcome


def attempt_delivery(client: httpx.Client, envelope: dict) -> str:
    headers = {
        "Content-Type": "application/json",
        "Idempotency-Key": envelope["id"],        # stable across every attempt
        "X-Delivery-Attempt": str(envelope["attempt"] + 1),
    }
    try:
        response = client.post(envelope["endpoint"], json=envelope["event"], headers=headers)
    except httpx.RequestError as exc:
        return reschedule(envelope, f"transport:{type(exc).__name__}")
    if response.status_code < 300:
        complete(envelope)
        return "delivered"
    if response.status_code in RETRYABLE_STATUS:
        return reschedule(envelope, f"http:{response.status_code}")
    complete(envelope)                             # 4xx is permanent: do not retry
    R.lpush(DEAD, envelope["id"])
    return "rejected"


def run_worker() -> None:
    running = True

    def stop(*_):
        nonlocal running
        running = False

    signal.signal(signal.SIGTERM, stop)
    timeout = httpx.Timeout(connect=CONNECT_TIMEOUT, read=READ_TIMEOUT, write=5.0, pool=5.0)
    with httpx.Client(timeout=timeout, limits=httpx.Limits(max_connections=64)) as client:
        while running:
            promote_due()
            reap_expired_leases()
            envelope = lease()
            if envelope is not None:
                attempt_delivery(client, envelope)


if __name__ == "__main__":
    run_worker()

Two details are load-bearing. promote_due and reap_expired_leases are called from the worker loop for brevity; in production run them in one or two dedicated schedulers, because every worker scanning the same sorted set every iteration is pure contention. And Idempotency-Key is the envelope ID, not a per-attempt UUID — regenerating it per attempt silently defeats consumer-side deduplication and is the most common way teams turn a duplicate-suppression system into a duplicate-delivery system.

What has to be validated before an envelope enters the queue

A delivery queue is a persistence layer holding customer data, often for days, and it is read by more services than the endpoint that produced it. Three controls matter.

Validate and canonicalise at enqueue, sign at dispatch. Reject malformed events before they are durable; a payload that cannot be serialised is far cheaper to fail at the producer than at attempt seven. But compute the signature in the worker, at the moment of the request, using the tenant’s current secret. Signing at enqueue freezes a secret into the message and breaks mid-flight rotation, so a queued backlog starts failing verification the moment a customer rotates a key.

Never store the secret in the message. Store a key reference (secret_version, kms_key_id) and resolve it at dispatch. Queue payloads land in broker snapshots, in dead-letter dumps, in the support tooling that a triage engineer screenshots into a ticket. Enable server-side encryption with a customer-managed key on every queue and on the dead-letter sink, and set retention explicitly rather than accepting the four-day default.

Bound the payload. SQS caps a message at 256 KB, and a webhook body plus headers plus delivery metadata gets there faster than teams expect. Enforce a limit at enqueue and, above it, use the claim-check pattern: put the body in object storage, put a pointer in the queue, and give the worker read access with a short-lived presigned URL. Silent truncation at the broker boundary produces payloads that fail signature verification at the consumer with no useful error anywhere.

Finally, the endpoint URL in the envelope is attacker-influenced data — a tenant chose it. It must be re-validated against your egress policy at dispatch time, not only at subscription time, because DNS for a hostname that resolved publicly last week can resolve to a link-local address today.

Operating the queue: metrics, deploys and capacity

Export four numbers per queue and treat them as the platform’s vital signs: ready depth, age of the oldest ready message, in-flight count, and promotion rate out of each delay tier. Age is the one to alert on. Depth is a function of both arrival rate and consumption rate and is therefore ambiguous; a ready message older than your delivery SLO means the pool cannot keep up, full stop. Tie the threshold to the target in defining SLOs for webhook delivery and derive the pool size from the arrival rate as described in sizing worker pools for webhook dispatch.

Deploys need a drain protocol. On SIGTERM a worker must stop leasing, finish or release in-flight envelopes, and only then exit — and the orchestrator’s grace period must exceed the lease window, or Kubernetes will SIGKILL workers mid-attempt and every rolling deploy will manufacture a small batch of duplicate deliveries. Set terminationGracePeriodSeconds above LEASE_SECONDS, not below it.

Queues themselves belong in infrastructure as code, with the redrive policy, retention, encryption key and delay-tier definitions in the same module as the workers that read them. Queue topology drifts silently otherwise: someone widens a visibility timeout during an incident and it stays widened for two years. Assert the topology in CI — a test that provisions the tiers against a local broker and checks that a failed delivery lands in the expected tier catches ladder misconfiguration long before a customer does.

Rolling out a delivery queue

1. Provision the ingest queue and retry ladder

Create the ready queue, every delay tier, and the dead-letter sink in one infrastructure module, with retention, encryption key and redrive policy set explicitly rather than defaulted. Name the tiers after their delay (retry-10s, retry-2h) so a dashboard is readable without a lookup table.

2. Pin the lease window to the HTTP timeout budget

Derive the visibility timeout from the client’s connect and read timeouts plus scheduling slack, and assert the relationship in a test so nobody can lower one without the other. This single setting decides your duplicate rate.

3. Separate first attempts from retry traffic

Put retries on their own queue with a bounded share of worker concurrency — typically 20% to 40%. Without the split, a large backlog of failing deliveries pushes time-to-first-attempt for brand-new events past your objective while throughput charts look normal.

4. Wire depth, age and in-flight metrics before launch

Export ready depth, oldest-message age, in-flight count and per-tier promotion rate on day one, and page on age. Retro-fitting these during an incident is how teams end up guessing which of the three queues is misbehaving.

5. Rehearse a drain and a redrive

In staging, stop the consumers, build a backlog of at least an hour of traffic, restart, and time the recovery. Then redrive the dead-letter sink and confirm the replayed envelopes carry their original idempotency keys. A topology that has never been drained is a topology whose recovery time is unknown.

Failure modes

Failure mode What you observe Mitigation
Lease shorter than the HTTP timeout budget Consumers report duplicate events; workers log “message no longer in flight” on ack Set the visibility timeout above connect + read timeout + slack; cap request time in the client
Delay ceiling below the backoff schedule Late retries fire far earlier than the ladder intends; retry storms during long outages Chain delay queues or use an external scheduler once backoff exceeds the broker’s per-message delay cap
Retry traffic sharing the ready queue with first attempts New events sit behind a backlog of failing ones; time-to-first-attempt spikes with no throughput change Give retries a separate queue and a bounded share of worker concurrency
Scheduler stopped, workers healthy Ready depth near zero, scheduled set growing without bound, deliveries never resume Alert on promotion rate and oldest scheduled score, not only on ready depth
In-flight ceiling reached Broker stops dispatching while depth stays high; no worker errors Shorten leases, reduce prefetch, or shard across queues to stay under the in-flight limit
Message body exceeds the broker limit Enqueue fails at the producer, or bodies are truncated and fail signature checks downstream Enforce a size cap at enqueue and switch to a claim check with object storage above it

Debugging checklist

Frequently Asked Questions

How many delay tiers does a retry ladder actually need?

Four to six is the usual sweet spot: enough to approximate an exponential curve, few enough that a dashboard stays readable. Each tier costs a broker object, a metric series and a line in the promotion logic, so tiers that differ by less than a factor of four buy you almost nothing. Pick the boundaries from your retry budget rather than from round numbers, so the last tier lands close to the point where you give up entirely.

Is it safe to purge a delay tier when it fills up during an incident?

Purging destroys customer events with no record, and a filling tier is almost never the problem you think it is. If you need the pressure off, stop the scheduler from promoting that tier instead: envelopes keep their due times, nothing is lost, and you can resume promotion at a throttled rate once the endpoint recovers. If envelopes genuinely have to leave the ladder, move them to the dead-letter sink so they remain replayable.

Should the scheduler run inside the workers or as its own deployment?

Run it as one or two dedicated replicas with a leader lease, not inside every worker. Promotion is a scan over shared state, so N workers scanning it produce N times the load for exactly the same result, and the contention grows as you scale the pool out. Two replicas with a lock give you failover without letting both promote the same envelope.

What happens to envelopes already queued when a tenant repoints or deletes a subscription?

Whatever your dispatcher resolves at attempt time wins, which is why the worker should look up the current subscription rather than trusting the URL frozen in the envelope. A deleted subscription should cause the worker to complete the envelope as cancelled and record that outcome, not to keep retrying a dead endpoint. Repointing mid-backlog is legitimate, but it means the tenant may receive old events at a new URL, so log the substitution explicitly.

How do I tell a healthy backlog of scheduled retries from a stalled scheduler?

Compare the lowest due timestamp in the scheduled set against the current clock. In a healthy system that gap is negative by only a second or two, because anything past due is promoted almost immediately regardless of how many millions of envelopes sit behind it. A gap that grows monotonically means promotion has stopped, and the size of the set tells you nothing on its own.

Does clock drift on the scheduler hosts affect the retry ladder?

Yes, because due times are absolute wall-clock timestamps written by one host and compared by another. A scheduler running a minute fast promotes short-tier envelopes before their backoff has elapsed, which quietly converts your ladder into a tighter retry loop against an endpoint that is already struggling. Keep NTP enforced on scheduler hosts and alert on offset, and never compute a due time from a monotonic clock that resets on restart.

Is a separate retry queue enough, or do retries need their own workers too?

A separate queue only helps if something bounds how much of the pool it can occupy. One pool polling both queues greedily will happily spend every slot on retries the moment the retry queue is deeper, which is the outcome the split was meant to prevent. Either run a second deployment with its own replica count, or make the poll loop weighted so retries can never claim more than their configured share of concurrency.