Choosing a Message Broker for Webhook Delivery

You are building the queue described in webhook delivery queue architecture and someone has to decide what sits underneath it. The candidate list is almost always the same four: Amazon SQS, RabbitMQ, Apache Kafka, Redis Streams. The comparisons you find are written for generic asynchronous work, and they optimise for the wrong things — throughput benchmarks and fan-out semantics — while ignoring the two properties that actually decide a webhook dispatcher’s fate: can the broker hold a message for two hours without a worker sitting on it, and can it keep one tenant’s backlog from becoming everyone’s backlog.

This guide works through that decision with the numbers that matter. The eventual worker count is a separate calculation covered in sizing worker pools for webhook dispatch; assume for now you know roughly how much concurrency you need.

Prerequisites

• Production metrics for at least 14 days: events produced per minute (p50 and peak), endpoint response time p50/p95/p99, and the failure rate by HTTP status class.
• A written retry policy — max attempts and the backoff ladder. If you do not have one, settle it first using choosing retry budgets and max attempts; the longest interval in that ladder is a hard input to this decision.
• A written ordering requirement, expressed as a key: “events for the same subscription must arrive in order” is a real requirement; “events should be ordered” is not.
• Python 3.11 with boto3 1.34, pika 1.3, confluent-kafka 2.4 and redis-py 5.0 for the prototypes, plus local containers for RabbitMQ 3.13, Kafka 3.7 and Redis 7.
• An owner named for broker operations. “The platform team” is not an owner; Kafka in particular will find out whether that name is real.

Step 1: Turn the delivery requirement into numbers

Four numbers decide most of this, and all four come from metrics you already have. Peak arrival rate is events per second at the busiest minute of the week, not the daily average. Retry amplification is how many extra messages the retry ladder produces per failed delivery — with an 8-attempt ladder and a 3% steady-state failure rate you are producing about 1.2 messages per event, but during an outage affecting 5% of your endpoints that same ladder produces closer to 1.4, and during a large provider outage it can double. Longest backoff interval is the top rung of the ladder; it decides whether the broker’s delay primitive is usable at all. Required in-flight count is concurrency, and it decides whether you will hit a broker ceiling.

"""Derive the four broker-selection inputs from production metrics."""
from dataclasses import dataclass


@dataclass
class DeliveryProfile:
    peak_events_per_sec: float      # busiest minute, not the daily mean
    p95_endpoint_seconds: float     # how long a slow endpoint holds a worker
    failure_rate: float             # fraction of attempts that are retryable
    max_attempts: int
    backoff_ladder_seconds: tuple[int, ...]

    @property
    def retry_amplification(self) -> float:
        """Expected messages produced per event, including retries."""
        total, probability = 1.0, self.failure_rate
        for _ in range(self.max_attempts - 1):
            total += probability
            probability *= self.failure_rate
        return total

    @property
    def peak_messages_per_sec(self) -> float:
        return self.peak_events_per_sec * self.retry_amplification

    @property
    def longest_delay_seconds(self) -> int:
        return max(self.backoff_ladder_seconds)

    @property
    def required_in_flight(self) -> float:
        """Little's Law: concurrency = arrival rate x service time."""
        return self.peak_messages_per_sec * self.p95_endpoint_seconds

    def report(self) -> str:
        return (
            f"peak messages/s      : {self.peak_messages_per_sec:,.0f}\n"
            f"retry amplification  : {self.retry_amplification:.2f}x\n"
            f"longest delay needed : {self.longest_delay_seconds}s\n"
            f"in-flight concurrency: {self.required_in_flight:,.0f}"
        )


if __name__ == "__main__":
    profile = DeliveryProfile(
        peak_events_per_sec=850,
        p95_endpoint_seconds=1.4,
        failure_rate=0.06,
        max_attempts=8,
        backoff_ladder_seconds=(10, 30, 120, 600, 1800, 3600, 7200),
    )
    print(profile.report())

For that profile the output is roughly 900 messages per second, a 7200-second longest delay, and about 1,270 in-flight messages. Two of those numbers already eliminate options: 7200 seconds is eight times SQS’s per-message delay ceiling, and 900 messages per second is above the 300 TPS an SQS FIFO queue sustains without batching. Neither is fatal, but both mean extra machinery, and you now know that before you write any code.

Step 2: Score the brokers on webhook-specific primitives

Throughput is the least interesting axis. Every one of these brokers will move a few thousand webhook messages per second on modest hardware, and your bottleneck is the remote HTTP endpoint anyway. What differs is the primitives.

The same comparison with the specifics that drive implementation work:

Read that table against Step 1’s numbers rather than in the abstract. A 7200-second top rung means SQS needs a chain (900s → 900s → 900s …) or an external scheduler; it means RabbitMQ needs a retry.2h queue with a message TTL and a dead-letter exchange pointing home; it means Kafka needs a consumer that can hold a record without committing for two hours, which in practice means a scheduler outside Kafka. Only Redis gets there with a single ZADD.

Step 3: Prototype the delayed retry path on the shortlist

Cut the list to two and build the same thing on both. The prototype should be exactly one contract — “hand me an envelope and a delay, give it back when the delay elapses, and survive a worker kill” — because that is the operation that every retry depends on and the one that is implemented differently everywhere.

Write the contract as a protocol first, then one adapter per candidate. Keeping the interface this narrow is also what makes Step 5 possible.

"""Delay-tier contract plus SQS and Redis adapters, for a like-for-like prototype."""
import json
import time
from typing import Protocol

import boto3
import redis


class DelayTier(Protocol):
    def schedule(self, envelope: dict, delay_seconds: float) -> None: ...
    def promote_due(self, limit: int = 100) -> list[dict]: ...


class SqsDelayTier:
    """SQS caps DelaySeconds at 900, so long delays hop through the tier repeatedly."""

    MAX_DELAY = 900

    def __init__(self, queue_url: str, client=None):
        self.queue_url = queue_url
        self.sqs = client or boto3.client("sqs")

    def schedule(self, envelope: dict, delay_seconds: float) -> None:
        remaining = max(0.0, delay_seconds)
        hop = min(self.MAX_DELAY, remaining)
        envelope = dict(envelope, remaining_delay=remaining - hop)
        self.sqs.send_message(
            QueueUrl=self.queue_url,
            MessageBody=json.dumps(envelope),
            DelaySeconds=int(hop),
        )

    def promote_due(self, limit: int = 10) -> list[dict]:
        """Messages become visible on their own; re-hop the ones still owed delay."""
        response = self.sqs.receive_message(
            QueueUrl=self.queue_url,
            MaxNumberOfMessages=min(limit, 10),
            WaitTimeSeconds=20,          # long poll: empty receives are billable
        )
        ready = []
        for message in response.get("Messages", []):
            envelope = json.loads(message["Body"])
            self.sqs.delete_message(
                QueueUrl=self.queue_url, ReceiptHandle=message["ReceiptHandle"]
            )
            if envelope.get("remaining_delay", 0) > 0:
                self.schedule(envelope, envelope["remaining_delay"])
            else:
                ready.append(envelope)
        return ready


class RedisDelayTier:
    """A sorted set keyed by due timestamp: exact delays, no ceiling."""

    def __init__(self, client: redis.Redis, key: str = "wh:scheduled"):
        self.r = client
        self.key = key

    def schedule(self, envelope: dict, delay_seconds: float) -> None:
        self.r.zadd(self.key, {json.dumps(envelope): time.time() + delay_seconds})

    def promote_due(self, limit: int = 100) -> list[dict]:
        now = time.time()
        due = self.r.zrangebyscore(self.key, "-inf", now, start=0, num=limit)
        if not due:
            return []
        # Only return entries this process actually removed, so two schedulers
        # racing on the same set cannot both promote the same envelope.
        pipe = self.r.pipeline()
        for raw in due:
            pipe.zrem(self.key, raw)
        removed = pipe.execute()
        return [json.loads(raw) for raw, ok in zip(due, removed) if ok]


def measure_scheduling_error(tier: DelayTier, delay: float, count: int) -> list[float]:
    """How late does each envelope actually arrive? Tail matters more than mean."""
    for index in range(count):
        tier.schedule({"id": index, "due": time.time() + delay}, delay)
    lateness, deadline = [], time.time() + delay + 60
    while len(lateness) < count and time.time() < deadline:
        for envelope in tier.promote_due():
            lateness.append(time.time() - envelope["due"])
        time.sleep(0.25)
    return sorted(lateness)

Run measure_scheduling_error with a few thousand envelopes at a 60-second delay and look at p99, not the mean. Redis lands within the scheduler’s tick interval. SQS is accurate to a second or two at low volume but the re-hop path adds a full extra visibility cycle per hop, so a 2-hour delay implemented as eight 900-second hops accumulates error at every one. RabbitMQ’s TTL queues have a sharper edge: TTL expiry is only evaluated at the head of the queue, so a message with a 10-second TTL sitting behind one with a 30-minute TTL does not dead-letter until the head does. If you build a single delay queue with per-message TTLs, you have built a queue that delivers retries in the wrong order and far too late. One queue per delay value is not an optimisation — it is the only correct way to do it.

Step 4: Price a bad month, not an average one

Every broker looks cheap at the average rate. The bill you should model is the one from the month a major consumer platform is down for six hours and your retry ladder is running at full amplification against 20% of your endpoints. For SQS, remember that polling is billable: an idle worker doing 20-second long polls generates about 130,000 requests a month on its own, and a pool of 200 workers doing 1-second polls generates 500 million.

"""Monthly cost model: request-priced (SQS) versus instance-priced (RabbitMQ, Kafka)."""
SECONDS_PER_MONTH = 30 * 24 * 3600
SQS_PER_MILLION = 0.40          # standard queue list price, us-east-1
SQS_FIFO_PER_MILLION = 0.50


def sqs_monthly_cost(
    messages_per_sec: float,
    worker_count: int,
    poll_wait_seconds: int = 20,
    receive_batch: int = 10,
    fifo: bool = False,
) -> dict:
    messages = messages_per_sec * SECONDS_PER_MONTH
    # One send, one receive per batch, one delete per message.
    work_requests = messages + (messages / receive_batch) + messages
    # Idle long polls still bill, whether or not they return anything.
    poll_requests = worker_count * (SECONDS_PER_MONTH / poll_wait_seconds)
    total = work_requests + poll_requests
    rate = SQS_FIFO_PER_MILLION if fifo else SQS_PER_MILLION
    return {
        "work_requests": work_requests,
        "poll_requests": poll_requests,
        "usd": total / 1_000_000 * rate,
    }


def broker_instance_cost(nodes: int, usd_per_hour: float, engineer_days: float) -> dict:
    infra = nodes * usd_per_hour * 24 * 30
    return {"infra_usd": infra, "toil_usd": engineer_days * 800, "usd": infra + engineer_days * 800}


if __name__ == "__main__":
    steady = sqs_monthly_cost(messages_per_sec=900, worker_count=120)
    outage = sqs_monthly_cost(messages_per_sec=2400, worker_count=300)
    print(f"SQS steady : ${steady['usd']:,.0f}")
    print(f"SQS outage : ${outage['usd']:,.0f}")
    print(f"RabbitMQ   : ${broker_instance_cost(3, 0.30, engineer_days=1.5)['usd']:,.0f}")
    print(f"Kafka      : ${broker_instance_cost(3, 0.55, engineer_days=4.0)['usd']:,.0f}")

The interesting output is not which number is smallest — it is which number moves. SQS cost scales with traffic, so a bad month costs three times a good one and the retry ladder’s amplification shows up directly on the invoice. A self-hosted broker costs the same in a bad month and more in engineering days, which never appear on the invoice at all. Put the engineer-days line in the model explicitly, because the argument otherwise reduces to comparing a real invoice against unpriced labour, and the labour always wins that argument until the on-call rotation starts.

Step 5: Commit to a recommendation and write down the exit cost

The decision collapses to four scenarios, and the honest answer is that most webhook platforms should be on SQS or Redis Streams unless something specific forces otherwise.

Kafka is right when the event log itself is the product — when you need to rebuild a consumer’s entire state from the beginning of time, or when webhook dispatch is one consumer group among many on an existing Kafka deployment. It is wrong as a dedicated webhook queue: it has no delay primitive, no dead-letter primitive, and a slow HTTP call inside the poll loop triggers consumer-group rebalances that stall every partition. If you use it, the HTTP call belongs on a thread pool with the poll loop committing offsets independently.

RabbitMQ is right when you need per-key ordering, you are not on AWS, and you already have people who operate it. The consistent-hash exchange gives per-tenant ordering with a bounded number of queues — the substrate that partitioning webhook queues by tenant builds on — and dead-lettering is genuinely first-class. Use quorum queues — classic mirrored queues have been deprecated and their failover behaviour is exactly what you do not want holding undelivered customer events.

SQS is right for the majority of webhook platforms on AWS. Its primitives map almost one-to-one onto the topology in the parent guide, the redrive policy gives you a dead-letter sink for free, and there is no broker to operate. Accept the 900-second ceiling and chain the ladder; it is a smaller cost than any of the alternatives here.

Redis Streams is right when scheduling precision matters, volumes are moderate, and Redis is already a hard dependency. Be clear-eyed about durability: with asynchronous replication, a failover can lose recently acknowledged writes, so an envelope confirmed to a producer may not survive. If that is unacceptable, pair it with the transactional outbox described in implementing the transactional outbox pattern for webhooks so the durable record lives in your database and Redis is only the dispatch buffer.

Whichever you pick, keep the adapter from Step 3 as the only place broker types appear. The exit cost is then one adapter, one conformance suite, and a dual-write window — not a rewrite of the dispatcher.

Verification and testing

Write the conformance suite once and run it against every adapter. Three properties matter, and all three are cheap to assert:

"""pytest conformance suite: run against each broker adapter you shortlist."""
import time

import pytest

from delivery.adapters import RedisDelayTier, SqsDelayTier  # your adapters


@pytest.fixture(params=["redis", "sqs"])
def tier(request, redis_client, sqs_queue_url):
    return RedisDelayTier(redis_client) if request.param == "redis" else SqsDelayTier(sqs_queue_url)


def test_envelope_is_not_promoted_early(tier):
    tier.schedule({"id": "a"}, delay_seconds=30)
    assert tier.promote_due() == []          # nothing may surface before it is due


def test_envelope_is_promoted_once_after_the_delay(tier):
    tier.schedule({"id": "b"}, delay_seconds=2)
    time.sleep(3)
    first = tier.promote_due()
    second = tier.promote_due()
    assert [e["id"] for e in first] == ["b"]
    assert second == []                      # no double promotion


def test_concurrent_schedulers_do_not_duplicate(tier):
    for index in range(50):
        tier.schedule({"id": f"c{index}"}, delay_seconds=1)
    time.sleep(2)
    promoted = [e["id"] for e in tier.promote_due(limit=50)] + \
               [e["id"] for e in tier.promote_due(limit=50)]
    assert len(promoted) == len(set(promoted)) == 50

Then verify the operational surface by hand before committing. On SQS, confirm the redrive policy is real rather than assumed:

aws sqs get-queue-attributes --queue-url "$READY_URL" \
  --attribute-names RedrivePolicy VisibilityTimeout DelaySeconds

# Watch delay-tier occupancy while a load test runs against a failing endpoint.
watch -n 5 'aws sqs get-queue-attributes --queue-url "$RETRY_15M_URL" \
  --attribute-names ApproximateNumberOfMessagesDelayed'

A shortlist candidate passes if the conformance suite is green, p99 scheduling error is under 10% of the shortest ladder rung, and a kill -9 on a worker mid-attempt results in redelivery rather than a lost envelope.

Failure modes and gotchas

• Benchmarking throughput instead of scheduling accuracy. Broker bake-offs almost always measure messages per second, which is the one axis where your webhook system is not constrained — the remote endpoint is. Measure delay fidelity, redelivery after a kill, and behaviour at the in-flight ceiling instead. A broker that does 200,000 messages/sec but cannot hold one for two hours is the wrong tool.
• Discovering the 900-second delay cap in production. DelaySeconds silently clamps to 900; a request for 7200 does not error, it just delivers 6,300 seconds early. The retry ladder then fires far ahead of schedule against an endpoint that is still down, burning the retry budget in minutes. Assert the requested delay against the broker’s ceiling in code, and chain hops explicitly as the adapter above does.
• Per-message TTL on a single RabbitMQ delay queue. Because TTL expiry is only evaluated at the head, a 10-second message queued behind a 30-minute one waits the full 30 minutes. The symptom is retries that arrive wildly late and out of order with no error anywhere. Use one queue per delay value with a queue-level x-message-ttl.
• Blocking the Kafka poll loop on an HTTP call. If processing exceeds max.poll.interval.ms (default 300,000 ms), the coordinator evicts the consumer and the group rebalances, which stalls every partition and redelivers records that were mid-flight. Under a widespread endpoint outage with long timeouts, this turns into a rebalance loop that stops delivery entirely. Dispatch HTTP work to a bounded executor and keep polling.
• Assuming Redis Streams durability matches a disk-backed broker. appendfsync everysec plus asynchronous replication means a failover can drop the last second of acknowledged writes. That is fine for a dispatch buffer fed from a durable outbox and unacceptable as the system of record for undelivered events. Decide which one it is before launch, not during a failover.

Frequently Asked Questions

Related

• Webhook Delivery Queue Architecture — the topology this broker has to implement.
• Partitioning Webhook Queues by Tenant — what per-key ordering support actually buys you.
• Sizing Worker Pools for Webhook Dispatch — turning in-flight requirements into a worker count.
• Per-Key Ordering With Partitioned Queues — the ordering guarantee behind the FIFO and partition columns.