Sizing Worker Pools for Webhook Dispatch

“How many workers should we run?” is usually answered with a number someone picked in the first week and doubled twice since. It is a question with an actual answer, and the arithmetic takes about ten minutes once you have measured the right quantity — which is not endpoint latency and not throughput, but the time a worker is occupied per delivery. Everything else follows from that number and the arrival rate the queue described in webhook delivery queue architecture is feeding you.

The second half of the question is the one that ruins deployments: what happens when you get the number wrong in the generous direction. An oversized pool does not sit idle. It converts your spare capacity into concurrent requests against other people’s servers, and the queue that was backing up quietly becomes a downstream incident with a 429 storm and a retry ladder amplifying it. This guide covers both directions — the sizing calculation, and the governor that keeps a big pool from being a weapon.

Prerequisites

• A histogram of per-attempt duration measured at the worker, covering at least a full weekly cycle. Percentiles alone are not enough; you need the mean, which most dashboards do not show by default.
• Arrival rate at the busiest minute of the week, including retry traffic. If you have not derived that, choosing a message broker for webhook delivery works through the amplification calculation.
• A written delivery latency objective — for example “p95 time from enqueue to first attempt under 5 seconds” — as in defining SLOs for webhook delivery. Without a target, sizing has no stopping condition.
• Python 3.11 with httpx 0.27 and asyncio, and a metrics pipeline that can export the age of the oldest ready message.
• Kubernetes 1.29 with KEDA 2.14 or an equivalent autoscaler that can read a queue metric rather than only CPU.

Step 1: Measure service time, not endpoint latency

Service time is the interval from the moment a worker leases an envelope to the moment it acknowledges it. That includes DNS resolution, TCP connect, the TLS handshake on a cold connection, request signing, the remote server’s processing time, response read, and the broker calls at both ends. Endpoint response time — the number your APM shows — is typically 60% to 80% of it, and sizing on it under-provisions by exactly that gap.

Instrument the whole span, and split it, because the components have very different fixes: a large connect and TLS share means you need connection reuse, not more workers.

"""Instrument true service time per delivery attempt, split by phase."""
import time
from contextlib import contextmanager

import httpx
from prometheus_client import Histogram

SERVICE_TIME = Histogram(
    "webhook_service_seconds",
    "Lease-to-acknowledge duration per attempt",
    buckets=(0.05, 0.1, 0.25, 0.5, 1, 2, 4, 8, 15, 30, 60),
)
PHASE = Histogram(
    "webhook_phase_seconds",
    "Duration of one phase of a delivery attempt",
    labelnames=("phase",),
    buckets=(0.001, 0.005, 0.02, 0.05, 0.1, 0.5, 1, 5, 30),
)


@contextmanager
def phase(name: str):
    started = time.perf_counter()
    try:
        yield
    finally:
        PHASE.labels(phase=name).observe(time.perf_counter() - started)


def deliver(client: httpx.Client, envelope: dict, secret: bytes, queue) -> int:
    """One attempt, fully instrumented. Returns the HTTP status code."""
    leased_at = time.perf_counter()
    try:
        with phase("sign"):
            headers = sign(envelope, secret)
        with phase("http"):
            response = client.post(
                envelope["endpoint"], content=envelope["body"], headers=headers
            )
        with phase("ack"):
            queue.ack(envelope["id"])
        return response.status_code
    finally:
        SERVICE_TIME.observe(time.perf_counter() - leased_at)


def summarise(samples: list[float]) -> dict[str, float]:
    """Little's Law needs the mean; the SLO conversation needs the tail."""
    ordered = sorted(samples)
    count = len(ordered)
    return {
        "mean": sum(ordered) / count,
        "p50": ordered[int(0.50 * count)],
        "p95": ordered[int(0.95 * count)],
        "p99": ordered[int(0.99 * count)],
    }

Expect the mean to be dominated by a small number of very slow endpoints. A population where p50 is 120 ms and p99 is 28 seconds routinely has a mean near 1.4 seconds, and it is the mean — not the p50 everyone quotes — that determines how many workers are busy at any instant. If the gap between mean and p50 is more than about 5x, capping request time is a cheaper intervention than adding workers; see handling slow webhook consumers.

Step 2: Apply Little’s Law to get the floor, then staff above it

Little’s Law states that the average number of items in a stable system equals the arrival rate multiplied by the average time each item spends in it. For a dispatch pool: the number of workers busy at any moment is arrival rate times mean service time. At 900 messages per second with a 1.4-second mean, 1,260 workers are busy at all times. That is a floor, not a target — provisioning exactly 1,260 means utilisation of 1.0, an unbounded queue, and a latency graph that goes vertical.

How much margin? The instinct is a fixed utilisation target like 70%, which would give 1,800 workers. That rule is badly calibrated at high concurrency: queueing systems get more efficient as they get larger, and the right margin scales with the square root of the offered load, not linearly with it. Square-root staffing says workers ≈ load + β√load, with β between 1 and 2 for interactive service levels. At 1,260 offered load that is 1,295 to 1,331 workers — a margin of 3% to 6%, not 43%. At small scale the same rule is far more conservative: 5 erlangs of load needs 7 to 9 workers, comfortably below 70% utilisation.

Rather than trust either heuristic, solve for the wait target directly with the Erlang C formula, computed in a numerically stable form so it does not overflow at four-figure worker counts:

"""Size a dispatch pool from arrival rate, service time and a queue-wait target."""
from math import ceil, sqrt


def erlang_b(servers: int, load: float) -> float:
    """Iterative Erlang B: stable for thousands of servers, unlike the factorial form."""
    blocking = 1.0
    for n in range(1, servers + 1):
        blocking = (load * blocking) / (n + load * blocking)
    return blocking


def erlang_c(servers: int, load: float) -> float:
    """Probability that an arriving delivery has to wait for a free worker."""
    if load >= servers:
        return 1.0
    blocking = erlang_b(servers, load)
    utilisation = load / servers
    return blocking / (1.0 - utilisation * (1.0 - blocking))


def mean_queue_wait(servers: int, load: float, service_time: float) -> float:
    if load >= servers:
        return float("inf")
    return erlang_c(servers, load) * service_time / (servers - load)


def size_pool(arrival_rate: float, service_time: float, target_wait: float) -> dict:
    load = arrival_rate * service_time              # Little's Law: busy workers
    servers = ceil(load) + 1
    while mean_queue_wait(servers, load, service_time) > target_wait:
        servers += 1
    return {
        "busy_workers_floor": load,
        "square_root_rule": ceil(load + 1.5 * sqrt(load)),
        "for_wait_target": servers,
        "utilisation": load / servers,
        "mean_wait_seconds": mean_queue_wait(servers, load, service_time),
    }


if __name__ == "__main__":
    for label, rate, mean in (("steady", 900, 1.4), ("outage surge", 2400, 3.1)):
        result = size_pool(rate, mean, target_wait=0.5)
        print(
            f"{label:<14} floor={result['busy_workers_floor']:>8,.0f} "
            f"sqrt={result['square_root_rule']:>6,} "
            f"target={result['for_wait_target']:>6,} "
            f"util={result['utilisation']:.1%}"
        )

Run it for both a normal peak and an outage surge — the surge row is the one that sets your autoscaler’s ceiling, because service time rises at the same moment arrival rate does. That interaction is what makes webhook pools feel non-linear: endpoints slow down, mean service time triples, and the same message rate now needs three times the workers to stand still.

Step 3: Split concurrency between processes and async tasks

“1,300 workers” is a concurrency number, not a process count. Webhook dispatch is almost pure IO wait: per attempt, real CPU is signing (tens of microseconds for HMAC-SHA256), JSON serialisation, and TLS record processing — call it 0.5 to 1 ms. At 930 deliveries per second that is under one CPU core for the entire fleet. Running 1,300 OS processes to do 0.6 cores of work is how teams end up paying for 40 pods to move 900 messages a second.

The layout that works: a small number of processes, each running hundreds of concurrent coroutines against a shared connection pool.

Three ceilings decide tasks per process, and none of them is CPU. Connection pool size — each in-flight request needs a connection, so max_connections must be at least the task count or tasks will block on the pool rather than on the network. File descriptors — the default 1,024 soft limit is reached quickly; raise it deliberately. Event-loop fairness — beyond roughly 500 tasks per loop, a single blocking call (a synchronous KMS lookup, a DNS resolution without a cache) stalls every other task on that loop, and tail latency degrades in a way that looks like network trouble.

"""Turn a concurrency target into a concrete pod, process and task layout."""
import asyncio
from dataclasses import dataclass
from math import ceil

import httpx


@dataclass
class Layout:
    total_concurrency: int
    tasks_per_process: int = 160        # keep well under event-loop fairness limits
    processes_per_pod: int = 4          # roughly one per vCPU for a 2 vCPU pod

    @property
    def processes(self) -> int:
        return ceil(self.total_concurrency / self.tasks_per_process)

    @property
    def pods(self) -> int:
        return ceil(self.processes / self.processes_per_pod)

    @property
    def file_descriptors_per_process(self) -> int:
        # one socket per task, plus broker connections, plus generous headroom
        return self.tasks_per_process * 2 + 256


async def dispatch_loop(queue, tasks: int) -> None:
    """One process: `tasks` concurrent attempts sharing one connection pool."""
    limits = httpx.Limits(max_connections=tasks, max_keepalive_connections=tasks)
    timeout = httpx.Timeout(connect=3.0, read=10.0, write=5.0, pool=5.0)
    semaphore = asyncio.Semaphore(tasks)

    async with httpx.AsyncClient(limits=limits, timeout=timeout) as client:
        async def run_one(envelope: dict) -> None:
            async with semaphore:
                try:
                    response = await client.post(
                        envelope["endpoint"], json=envelope["event"],
                        headers={"Idempotency-Key": envelope["id"]},
                    )
                    await queue.settle(envelope, response.status_code)
                except httpx.RequestError as exc:
                    await queue.settle(envelope, None, error=str(exc))

        pending: set[asyncio.Task] = set()
        while True:
            envelope = await queue.lease()
            if envelope is None:
                await asyncio.sleep(0.05)
                continue
            task = asyncio.create_task(run_one(envelope))
            pending.add(task)
            task.add_done_callback(pending.discard)
            if len(pending) >= tasks:
                await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)


if __name__ == "__main__":
    layout = Layout(total_concurrency=1300)
    print(f"processes={layout.processes} pods={layout.pods} "
          f"ulimit -n >= {layout.file_descriptors_per_process}")
    asyncio.run(dispatch_loop(queue=..., tasks=layout.tasks_per_process))

For 1,300 concurrency that is 9 processes across 3 pods — a very different infrastructure bill from 1,300 threads. Keep one connection pool per process rather than per task: keep-alive reuse is what removes the TLS handshake from service time, and Step 1’s phase histogram will show the improvement immediately as the http phase mean drops.

Step 4: Autoscale on queue age rather than queue depth

Depth is a stock; it tells you how much work exists, not whether anyone is waiting too long. Two systems with a depth of 5,000 are in completely different states if one drains it in four seconds and the other in nine minutes. Age of the oldest ready message is a direct measurement of the thing your objective is written about, and it responds immediately when service time degrades — which depth does not, since a slowdown initially shows up as flat depth with rising latency.

In practice, drive the replica count from age with a proportional controller and clamp the step size. KEDA can do this directly against ApproximateAgeOfOldestMessage on SQS or a Prometheus query on Redis; the logic it applies is what the controller below expresses explicitly, which is worth understanding before you tune the thresholds.

"""Replica controller: proportional on oldest-message age, damped and clamped."""
from dataclasses import dataclass


@dataclass
class ScalingPolicy:
    target_age_seconds: float = 30.0
    min_replicas: int = 3
    max_replicas: int = 60
    max_step_up: int = 8          # never more than +8 replicas per interval
    max_step_down: int = 1        # scale in one replica at a time
    scale_in_below: float = 10.0  # hysteresis gap under the target


def desired_replicas(
    current: int, oldest_age: float, arrival_rate: float,
    service_time: float, tasks_per_replica: int, policy: ScalingPolicy,
) -> int:
    # Floor from Little's Law: never drop below what steady state requires.
    floor = max(policy.min_replicas,
                -(-int(arrival_rate * service_time) // tasks_per_replica))

    if oldest_age > policy.target_age_seconds:
        ratio = oldest_age / policy.target_age_seconds
        proposed = min(current + policy.max_step_up, int(current * min(ratio, 2.0)) + 1)
    elif oldest_age < policy.scale_in_below:
        proposed = current - policy.max_step_down
    else:
        proposed = current                       # inside the deadband: do nothing

    return max(floor, min(policy.max_replicas, proposed))

Two operational rules matter more than the controller’s shape. Scale-in must never be faster than the lease window: terminating a pod holding 160 in-flight attempts either loses them for the length of the visibility timeout or duplicates them, so keep terminationGracePeriodSeconds above the lease and drain on SIGTERM. And the floor is not zero. Scaling a webhook dispatcher to zero looks efficient right up to the moment a burst arrives and the first pod has to cold-start, resolve DNS, and warm a connection pool while age climbs past the objective.

Step 5: Cap concurrency per endpoint before adding workers

Here is the part that sizing calculations miss: your worker pool is not the only capacity in the system. Doubling workers doubles the concurrent requests arriving at consumer endpoints, and consumers are usually smaller than you are. A pool sized for your backlog is a load test against every customer simultaneously — and their 429s and timeouts feed straight back into your retry ladder, raising both arrival rate and mean service time. That is the loop that turns a 20-minute backlog into a three-hour one because you scaled up.

The fix is a per-endpoint concurrency limit that adapts to observed behaviour, applied at lease time so a slow endpoint cannot occupy an unbounded share of the pool:

"""AIMD per-endpoint concurrency governor: additive increase, multiplicative decrease."""
import threading
import time


class EndpointGovernor:
    """Bounds concurrent in-flight attempts per endpoint and adapts on feedback."""

    def __init__(self, initial: int = 4, ceiling: int = 64, floor: int = 1):
        self.limits: dict[str, float] = {}
        self.in_flight: dict[str, int] = {}
        self.retry_after: dict[str, float] = {}
        self.initial, self.ceiling, self.floor = initial, ceiling, floor
        self.lock = threading.Lock()

    def try_acquire(self, endpoint: str) -> bool:
        with self.lock:
            if time.monotonic() < self.retry_after.get(endpoint, 0.0):
                return False                       # honouring Retry-After
            limit = self.limits.setdefault(endpoint, float(self.initial))
            current = self.in_flight.get(endpoint, 0)
            if current >= int(limit):
                return False
            self.in_flight[endpoint] = current + 1
            return True

    def release(self, endpoint: str, status: int | None, retry_after: float = 0.0) -> None:
        with self.lock:
            self.in_flight[endpoint] = max(0, self.in_flight.get(endpoint, 1) - 1)
            limit = self.limits.get(endpoint, float(self.initial))
            if status is not None and 200 <= status < 300:
                self.limits[endpoint] = min(self.ceiling, limit + 0.5)      # additive up
            elif status in (429, 503) or status is None:
                self.limits[endpoint] = max(self.floor, limit * 0.5)        # multiplicative down
                if retry_after:
                    self.retry_after[endpoint] = time.monotonic() + retry_after

    def snapshot(self) -> list[tuple[str, int, int]]:
        with self.lock:
            return sorted(
                ((endpoint, int(limit), self.in_flight.get(endpoint, 0))
                 for endpoint, limit in self.limits.items()),
                key=lambda row: row[1],
            )

With this in place the pool size question changes character: extra workers now improve throughput for healthy endpoints and are simply refused entry to unhealthy ones. Pair it with the ingest-side controls in applying backpressure to webhook consumers, and with per-tenant caps if one customer’s endpoints dominate your traffic — the mechanics of that split are in partitioning webhook queues by tenant.

Verification and testing

Assert the sizing maths, then validate the layout under load. The calculator has properties worth pinning: more workers must never increase predicted wait, and the pool must always exceed the Little’s Law floor.

"""Regression tests for the sizing calculator and the endpoint governor."""
import pytest

from delivery.sizing import EndpointGovernor, mean_queue_wait, size_pool


def test_wait_falls_monotonically_as_workers_are_added():
    load, service = 120.0, 1.2
    waits = [mean_queue_wait(servers, load, service) for servers in range(121, 200)]
    assert all(later <= earlier for earlier, later in zip(waits, waits[1:]))


def test_pool_always_exceeds_the_littles_law_floor():
    result = size_pool(arrival_rate=900, service_time=1.4, target_wait=0.5)
    assert result["for_wait_target"] > result["busy_workers_floor"]
    assert 0.9 < result["utilisation"] < 1.0        # large pools run hot and still meet the SLO


def test_saturated_pool_reports_infinite_wait():
    assert mean_queue_wait(servers=100, load=100.0, service_time=1.0) == float("inf")


def test_governor_halves_the_limit_on_429():
    governor = EndpointGovernor(initial=16)
    assert governor.try_acquire("https://example.test/hook")
    governor.release("https://example.test/hook", status=429, retry_after=0)
    assert dict(( e, l) for e, l, _ in governor.snapshot())["https://example.test/hook"] == 8


def test_governor_refuses_beyond_the_limit():
    governor = EndpointGovernor(initial=2)
    url = "https://example.test/hook"
    assert governor.try_acquire(url) and governor.try_acquire(url)
    assert not governor.try_acquire(url)

Then validate against reality, because the calculator’s inputs are estimates and the only honest check is a controlled load test. Replay a recorded peak against a sandbox endpoint at 1x, 2x and 4x, and record the age of the oldest ready message at each level:

# SQS: the single number that decides whether the pool is big enough.
aws cloudwatch get-metric-statistics --namespace AWS/SQS \
  --metric-name ApproximateAgeOfOldestMessage \
  --dimensions Name=QueueName,Value=webhook-ready \
  --start-time "$(date -u -d '30 minutes ago' +%FT%TZ)" \
  --end-time "$(date -u +%FT%TZ)" --period 60 --statistics Maximum

# Redis: same signal, computed from the envelope's enqueue timestamp.
redis-cli --raw lindex wh:ready -1 | python -c \
 "import json,sys,time; print(round(time.time()-json.load(sys.stdin)['enqueued_at'],1),'s')"

The pool is correctly sized when oldest-message age stays under the objective at 2x peak, mean utilisation sits in the 85% to 95% band rather than at 40%, and scaling from 1x to 2x traffic does not increase the downstream 429 rate — that last one is the evidence that Step 5’s governor is doing its job.

Failure modes and gotchas

• Sizing on p50 endpoint latency. Little’s Law takes the mean of the full service time, and in a webhook population the mean is dragged far above the median by a minority of endpoints that time out. Sizing on a 120 ms p50 when the mean is 1.4 seconds under-provisions the pool by more than 10x, and the symptom is a queue that is fine all week and collapses at every peak.
• Autoscaling on depth with no age signal. Depth-based rules oscillate because depth falls quickly once workers catch up, triggering an immediate scale-in that recreates the backlog. Worse, depth stays flat when endpoints slow down, so the autoscaler does nothing during the exact incident it was built for. Scale on age, keep depth as a dashboard metric.
• Scale-in shorter than the lease window. If pods are terminated faster than in-flight attempts complete, every scale-in event produces either duplicate deliveries (lease expiry mid-flight) or a stall equal to the visibility timeout. Set the grace period above the lease, drain on SIGTERM, and never let node autoscaling evict dispatch pods without the same protection.
• Concurrency without file descriptor and pool headroom. At 160 tasks per process against a default 1,024 descriptor limit and a default max_connections of 100, most tasks queue on the connection pool instead of the network. The tell is that added workers do not increase throughput and pool-wait time appears in the phase histogram. Raise both limits together with the task count.
• Solving a consumer-side bottleneck with more workers. If the backlog exists because endpoints are slow rather than because you lack capacity, extra workers increase concurrent pressure on those endpoints, raise their latency further, and grow mean service time — so the same pool now needs even more workers. Cap per-endpoint concurrency first and confirm the bottleneck is yours before scaling.

Frequently Asked Questions

Related

• Webhook Delivery Queue Architecture — the queue these workers consume from.
• Partitioning Webhook Queues by Tenant — dividing a sized pool fairly between customers.
• Choosing a Message Broker for Webhook Delivery — in-flight ceilings that cap how large a pool can usefully get.
• Tracking Webhook Delivery Latency Percentiles — measuring the latency this sizing is meant to protect.