Designing webhook fan-out architectures

A single domain event — an order placed, a payment settled — often needs to reach many subscribers at once. This guide builds a fan-out architecture where one inbound event is delivered to every interested endpoint through per-subscriber queues and isolated workers, so one slow or failing subscriber cannot stall delivery to the others. It builds on sync vs async webhooks and complements when to use synchronous callbacks vs async webhooks, which establishes why fan-out delivery must be asynchronous in the first place.

The defining hazard of fan-out is head-of-line blocking: if all subscribers share one queue and one endpoint hangs for 30 seconds per request, every other subscriber waits behind it. Isolating each subscriber onto its own queue with its own workers removes that coupling — a dead endpoint backs up only its own lane.

Prerequisites

• Python 3.10+ with redis and rq (Redis Queue) installed, plus a reachable Redis instance.
• A subscriptions table or store mapping event types to subscriber endpoints and secrets.
• An understanding of why this work is asynchronous, covered in when to use synchronous callbacks vs async webhooks.
• A prior decision that push delivery is the right transport for these subscribers; webhooks vs polling vs WebSockets covers when a pull model costs less to run.
• A retry policy in mind; this design defers the backoff details to resilient delivery & retry strategies.

Step 1: Persist the event once and return immediately

The ingest endpoint does the minimum: validate, store the event durably, and respond 202 Accepted. It must never block on delivery, because delivery to N subscribers can take arbitrarily long.

import json
import uuid
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
import redis

app = FastAPI()
r = redis.Redis(decode_responses=True)

@app.post("/events")
async def ingest(request: Request):
    payload = await request.json()
    event_id = str(uuid.uuid4())
    event = {"id": event_id, "type": payload["type"], "body": payload}
    # Store the canonical event once; deliveries reference it by id.
    r.set(f"event:{event_id}", json.dumps(event))
    fan_out(event)  # enqueue jobs; does not perform HTTP delivery
    return JSONResponse({"event_id": event_id}, status_code=202)

Step 2: Fan out to per-subscriber queues

Look up every subscriber for the event type and enqueue one delivery job per subscriber onto that subscriber’s own queue. The queue name is keyed by subscriber id, which is what gives each one an isolated lane.

from rq import Queue

def subscribers_for(event_type: str) -> list[dict]:
    # In production this reads your subscriptions store.
    raw = r.smembers(f"subs:{event_type}")
    return [json.loads(s) for s in raw]

def fan_out(event: dict) -> None:
    for sub in subscribers_for(event["type"]):
        # One queue per subscriber => no shared head-of-line.
        q = Queue(f"deliver:{sub['id']}", connection=r)
        q.enqueue(
            "delivery.deliver",          # worker function path
            event_id=event["id"],
            subscriber=sub,
            job_timeout=30,              # cap a single attempt
            retry=None,                  # retries handled explicitly below
        )

The queue name is the whole isolation mechanism, so it is worth being explicit about what each part of it does and what the job actually carries. The job holds an event id rather than the event body: the payload stays in one place, and a queued job that is retried tomorrow still signs exactly the bytes that were stored today.

Step 3: Deliver from isolated workers

The delivery function loads the stored event, signs it, and POSTs to the subscriber. Each subscriber’s queue is drained by its own worker process, so a hung endpoint consumes only its lane’s worker, never another subscriber’s.

# delivery.py
import hashlib
import hmac
import json
import httpx
import redis

r = redis.Redis(decode_responses=True)

def sign(secret: str, body: bytes) -> str:
    return hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()

def deliver(event_id: str, subscriber: dict) -> str:
    event = json.loads(r.get(f"event:{event_id}"))
    body = json.dumps(event["body"]).encode()
    headers = {
        "Content-Type": "application/json",
        "X-Event-Id": event_id,                 # stable id => consumer can dedupe
        "X-Signature": sign(subscriber["secret"], body),
    }
    state_key = f"delivery:{event_id}:{subscriber['id']}"
    r.hincrby(state_key, "attempts", 1)
    try:
        resp = httpx.post(subscriber["url"], content=body, headers=headers, timeout=10)
        resp.raise_for_status()
    except httpx.HTTPError as exc:
        r.hset(state_key, "status", "failed")
        raise  # re-raise so the queue's retry/DLQ policy can act
    r.hset(state_key, "status", "delivered")
    return "delivered"

Run one worker pool per subscriber lane. With RQ you point a worker at the specific queues it should drain:

# A worker dedicated to subscriber sub_42's lane.
rq worker deliver:sub_42 --url redis://localhost:6379

Two settings decide whether that worker behaves under stress. The first is the relationship between job_timeout and the HTTP timeout: the job deadline must be comfortably larger, because it has to cover DNS resolution, the TLS handshake, the request itself, and the bookkeeping around it. A sensible default is roughly three times the HTTP timeout — timeout=10 with job_timeout=30. Invert them and the worker is killed mid-request, which leaves the delivery record stuck in sending forever with no exception ever reaching the except branch. The observable symptom is a lane whose attempt counter climbs while its status field never changes, and it is easy to misread as a consumer problem when it is entirely self-inflicted.

The second is connection reuse. Constructing httpx.post(...) per delivery opens a fresh TCP connection and repeats the TLS handshake every time, which adds 80–150 ms to every attempt and accumulates sockets in TIME_WAIT — at a few hundred deliveries per second per host you will exhaust the ephemeral port range and start seeing connection failures that look like the subscriber refusing traffic. Build one httpx.Client per worker process at startup, give it a keep-alive pool sized to the lane’s concurrency, and reuse it for every job. On a warm connection the same delivery often completes in a third of the time, which directly reduces how long a slow subscriber occupies its lane.

The payoff shows up on a time axis. With job_timeout=30 on a subscriber whose endpoint hangs, that lane burns its full timeout and then retries, while the other lanes are already finished — the same event, three completely different completion times.

Step 4: Track per-subscriber delivery state

Because each subscriber is independent, delivery state is per (event_id, subscriber_id): attempts, last status, and whether it has been dead-lettered. This lets you retry or replay a single subscriber without touching the others, and lets you answer “who got event X?” precisely. Route a subscriber to a dead-letter queue only after its own attempts are exhausted; the other subscribers’ deliveries are unaffected.

Because that record is per subscriber rather than per event, each subscriber runs its own small state machine over the same event. Subscriber A can be sitting in failed awaiting a retry while subscriber B is already delivered, and dead-lettering A leaves B’s record untouched.

Sizing lanes: dedicated queues versus hashed shards

A queue per subscriber is the cleanest form of isolation and the least scalable. Redis itself is untroubled by tens of thousands of list keys, but each lane needs a worker to drain it, and an idle RQ worker still costs 30–80 MB of resident memory plus a Redis connection. At 5,000 subscribers that is 5,000 processes and roughly 250 GB of memory doing almost nothing — the design collapses long before Redis does. The break-even point in practice sits in the low hundreds of subscribers: below it, dedicated lanes are simple and worth it; above it, hash subscribers into a fixed number of shard lanes and accept isolation that is approximate rather than absolute.

Sharding is a one-line change to the lane function, but the hash choice matters more than it looks. Python’s built-in hash() is salted per process unless PYTHONHASHSEED is pinned, so the same subscriber lands in different buckets on different workers and after every restart — which silently destroys both per-lane ordering and any monitoring keyed on the lane. Use an explicit, stable digest:

import hashlib

SHARD_COUNT = 64

def lane_for(subscriber_id: str) -> str:
    """Stable across processes, restarts and interpreter versions."""
    digest = hashlib.blake2b(subscriber_id.encode(), digest_size=8).digest()
    bucket = int.from_bytes(digest, "big") % SHARD_COUNT
    return f"deliver:shard{bucket:02d}"

Pick SHARD_COUNT from your worker budget rather than your subscriber count, and set it once: a change re-maps every subscriber, so jobs already queued under the old scheme keep their old lane while new jobs go elsewhere, and any ordering guarantee is void for the duration. If you expect to grow, start with a count comfortably above what you need (64 or 128 lanes is cheap) rather than resharding later. The remaining exposure is a noisy bucket: if a large subscriber and a chronically slow one hash together, the slow one delays the large one. Give known-slow endpoints an explicit override to their own dedicated lane and let the hash handle everyone else — the override list stays short precisely because slow endpoints are rare.

What to watch per lane

Fan-out monitoring fails in a specific way: fleet-wide aggregates look perfect while an individual lane is entirely stuck, because a stalled lane produces no failures — it produces nothing at all. Every signal below is therefore computed per lane and alerted per lane.

The last row is the one worth building first. It is the only signal that catches a partially completed fan-out, and a partial fan-out is invisible from every other angle: the event is stored, the deliveries that were enqueued all succeed, and the dashboard is green while a subscriber never learns the event happened.

Verification

A unit test should confirm that one ingested event produces exactly one job per matching subscriber, on distinct queues.

from rq import Queue
import fakeredis, json

def test_fan_out_enqueues_one_job_per_subscriber():
    fake = fakeredis.FakeStrictRedis(decode_responses=True)
    fake.sadd("subs:order.created",
              json.dumps({"id": "a", "url": "http://a", "secret": "s"}),
              json.dumps({"id": "b", "url": "http://b", "secret": "s"}))
    # ... wire fan_out to `fake`, then:
    event = {"id": "e1", "type": "order.created", "body": {}}
    fan_out(event)
    assert Queue("deliver:a", connection=fake).count == 1
    assert Queue("deliver:b", connection=fake).count == 1

To prove isolation, point one subscriber at a sink that sleeps and confirm the other still receives promptly:

# Slow endpoint for sub A; healthy endpoint for sub B.
curl -fsS -X POST localhost:8000/events \
  -H 'content-type: application/json' \
  --data '{"type":"order.created","order_id":"ord_1"}'
# sub B's worker should mark "delivered" while sub A is still retrying.

Failure modes and gotchas

• Shared queue defeats the whole design. If you enqueue all deliveries onto one queue and merely tag them with a subscriber id, a single slow endpoint blocks the head of the line for everyone. The queue name must be per subscriber.
• Unbounded queue-per-subscriber sprawl. Tens of thousands of subscribers means tens of thousands of queues and workers. Above a few hundred, shard subscribers across a fixed pool of N queues by hash(subscriber_id) % N so isolation is approximate but worker count stays bounded.
• Re-fetching a mutated event on retry. Workers load the event by id; if some other process mutates event:<id> between attempts, retries deliver different bytes and break signatures and consumer dedupe. Treat the stored event as immutable once written.
• Enqueue after a partial crash. If ingest stores the event but crashes before all jobs are enqueued, some subscribers never receive it. Make fan-out idempotent and re-runnable from the stored event so a sweeper can re-enqueue missing (event_id, subscriber_id) pairs.
• Fanning out inline for a broadcast event. Enqueueing is fast per job but not free: at roughly 0.4 ms of round trip per enqueue, a broadcast to 20,000 subscribers spends eight seconds inside the request handler, long past the client’s timeout. The client then retries an event that was already stored, and you fan the same event out twice. Enqueue a bounded first batch inline — a few hundred jobs — and hand the remainder to a single background fan-out job that resumes from the last subscriber id it processed.
• Delivering to a subscriber that no longer exists. Jobs queued before a subscriber was deleted or disabled will still fire, because the subscriber record was copied into the job at enqueue time. An endpoint that was revoked for a security reason keeps receiving events until its lane drains, which is the worst possible time for that to be true. Re-read the subscription by id at delivery time and treat a missing or disabled record as a terminal, non-retryable outcome rather than a failure to retry.
• Re-serialising the payload before signing. If the delivery worker rebuilds the JSON with json.dumps while the consumer verifies against the bytes it received, any difference in key order, whitespace, or unicode escaping produces a signature mismatch that looks exactly like a wrong secret. Store the canonical body as bytes at ingest, sign those bytes, and never let an intermediate layer pretty-print or re-encode them.
• Synchronised retries when an endpoint recovers. A subscriber that was down for ten minutes has a lane full of jobs whose backoff timers all expire within the same second, so recovery is greeted with a burst that knocks it over again — and the second failure has a longer backoff, so the burst returns bigger. Add proportional jitter to every retry delay and cap the lane’s concurrency during recovery so the first minute after a breaker closes is a trickle rather than a flood.

Frequently Asked Questions

Related

• When to use synchronous callbacks vs async webhooks
• Webhooks vs polling vs WebSockets
• Sync vs async webhooks
• Dead-letter queue architecture