Webhook Timeout and Connection Management for High-Volume Dispatchers

Everything else in Resilient Delivery & Retry Strategies assumes an attempt eventually finishes — and the component that decides when an attempt finishes is the HTTP client inside your dispatcher, not the retry policy above it. Retry schedules, breakers and dead-letter routing all act on an outcome. Timeout and connection management is the layer that manufactures that outcome: it turns an open socket to an unresponsive consumer into a bounded, classifiable failure within a known number of milliseconds. Get it wrong and there is nothing for the rest of the stack to react to, because the worker is still sitting in recv() waiting for a response that will never come.

A dispatcher fanning events out to thousands of independently operated consumer endpoints is one of the harshest HTTP client workloads there is. You do not control the peers, their TLS stacks, their idle timeouts, their DNS records or their capacity. Some answer in 40 ms; some answer in 25 seconds; some accept the TCP connection and then never send a byte. The client configuration that survives that mix looks nothing like the defaults shipped by requests, httpx or net/http.

The time budget of a single delivery attempt

“Timeout” is not one number. A delivery attempt is a sequence of distinct phases, each with its own failure signature, and lumping them into one deadline destroys the information you need to act. DNS resolution, TCP connect and the TLS handshake are setup: they either complete in tens of milliseconds or something is structurally wrong. Writing the request body is bounded by payload size and the peer’s receive window. Then comes the part you cannot predict — the consumer’s own processing — followed by reading the response.

Two consequences follow. First, the setup phases justify a tight, aggressive deadline — a consumer that has not completed TLS in three seconds is not going to complete it in thirty. Second, because setup is ~150 ms of pure overhead per attempt, reusing connections is not a micro-optimisation: at 4,000 deliveries per second it is the difference between a comfortable fleet and one that is 30% occupied doing handshakes. That case is worked through in connection pooling and keep-alive for webhook senders.

Pattern 1: layered timeouts instead of one global deadline

The most common production mistake is timeout=30 — a single scalar applied to the whole request. It fails in both directions at once. Thirty seconds is far too generous for a connect phase, so a black-holed endpoint pins a worker for half a minute before anything notices. And it is often too tight for a legitimately slow but healthy consumer that batches work, so you abort successful deliveries and retry them, creating duplicates.

Layered timeouts assign each phase its own budget:

The retry-safety column is the one that matters architecturally. A connect timeout is a clean, idempotent-by-construction failure — retry it immediately with a fresh attempt. A read timeout is ambiguous: the consumer may have committed a database transaction and simply been slow to answer. That ambiguity is what forces idempotency in webhooks on the consumer side, and it is why read timeouts should be counted separately in your metrics rather than folded into a generic delivery_failed counter. Choosing the actual numbers from measured endpoint behaviour is covered in setting connect and read timeouts.

Deriving each budget from measured behaviour rather than folklore

The values in that table are defaults, and defaults are where you start, not where you finish. Each budget has a derivation, and running it once against your own telemetry usually moves at least one number substantially.

The connect budget derives from network physics. A TLS 1.3 handshake costs one round trip after the TCP handshake’s one, so a peer 40 ms away needs about 80 ms of setup and a peer 250 ms away about 500 ms. Adding DNS — often 20 ms warm, occasionally 2 seconds when a resolver is failing over — a 3-second connect budget covers a transcontinental peer roughly six times over. That headroom is the point: connect timeouts should never fire on a healthy endpoint, so a rising connect_timeout rate is unambiguous evidence about the peer rather than a signal that your budget is tight.

The read budget derives from the consumer’s own latency distribution, and the right statistic is p99 rather than p50 or max. A budget at 2–3× the observed p99 tolerates ordinary variance without waiting on genuinely stuck requests. Worked through: an endpoint with p50 180 ms, p95 900 ms and p99 2.1 s gets a read timeout of 5–6 seconds. Setting it at the p50 would abort a quarter of successful deliveries; setting it at 30 seconds means a stuck request occupies its slot for fifteen times longer than any healthy one ever needs. Recompute these per endpoint weekly and store the result on the endpoint record — consumer latency moves when consumers deploy, and a timeout tuned to last quarter’s behaviour is how a routine consumer-side change becomes your incident.

Two constraints bound the whole calculation from above. The sum of connect, write and read must sit comfortably below the queue’s visibility timeout, or the broker will redeliver a message whose first attempt is still in flight and you will manufacture duplicates that look exactly like a consumer bug. And the total attempt budget multiplied by the attempt count must fit inside the delivery window you promise; a 20-second attempt budget with six attempts spends two minutes on timeouts alone, which for a one-minute delivery promise is arithmetic that cannot work no matter how the backoff is tuned.

One refinement is worth the complexity at scale: derive the read timeout adaptively, as a rolling p99 over the last few hundred deliveries per endpoint, clamped to a sane range such as 3–15 seconds. Adaptive budgets track consumers who legitimately get slower under their own peak load, which is exactly when a static budget starts aborting successful work and converting a busy consumer into a failing one. Clamp both ends, and never let the adaptive value be learned from a period in which the endpoint was failing, or a degraded consumer teaches the dispatcher to be infinitely patient with it.

Pattern 2: per-host connection pools with bounded leases

A dispatcher’s HTTP client holds one global pool by default. That single shared resource is where cross-tenant blast radius enters the system: connections are leased on demand, and a host that holds each lease for 25 seconds will accumulate leases far faster than a host that holds them for 200 ms. Left unbounded, the slowest endpoint in your customer base ends up owning most of the pool.

The fix is a two-level bound: a global ceiling that protects file descriptors and memory, and a per-host ceiling that caps how much of the global pool any single destination can occupy.

Per-host caps also change how you size the pool overall. The useful ratio is not connections-to-workers but connections-to-hosts-in-flight. If 32 workers are spread across 400 active destinations, a 200-connection pool is generous; if all 32 are hammering one destination during a replay, a per-host cap of 8 will deliberately make 24 of them wait — which is the correct behaviour, and why the pool-acquire timeout must be short enough to shed rather than silently queue. Pool sizing against fleet-wide worker counts is developed further in sizing worker pools for webhook dispatch.

The lifecycle of a pooled connection

Pool sizing is only half the problem; the other half is knowing which states a pooled connection can be in and which of those states is lying to you. A connection is created, becomes idle in the pool, is leased for a request, returns to idle, and is eventually expired by your keepalive_expiry or closed by the peer. The dangerous state is the one you cannot observe: closed by the peer but still listed as idle in your pool. TCP gives the client no notification until it writes, so the failure surfaces as an error on the first byte of the next request — attributed, wrongly, to the consumer.

Setting keepalive_expiry below the shortest peer idle timeout you observe is the primary defence, and 30 seconds is safe against the common defaults — nginx ships 75 seconds, many cloud load balancers 60, and Go’s net/http server 90. Where a peer advertises Keep-Alive: timeout=N, honour it per host. A secondary defence is to make one retry on a write failure that occurs before any response bytes arrive: that specific failure is provably safe to repeat because the request never reached the application, and treating it as a first-class retry rather than a delivery failure removes an entire category of phantom errors from your dashboards.

The remaining cause of pool churn is DNS. A pooled connection pins the address it resolved at creation, so a consumer that moves behind a short-TTL record keeps receiving traffic at the old address until every pooled connection expires. Bounding keepalive_expiry bounds that staleness too, which is the second reason the value should be tens of seconds rather than minutes. Watch connection_reuse_ratio per host as the summary statistic: a healthy busy endpoint sits above 0.9, and any sustained drop means connections are being created faster than they are being reused, which is always either a peer-side timeout change, a DNS change, or a code path building clients per request.

Pattern 3: bulkheads that keep one endpoint off the whole worker pool

Connection caps bound sockets. They do not, on their own, bound workers. A worker that is blocked waiting for a pool lease is just as unavailable as one blocked reading a response. To protect the worker pool you need an admission gate in front of the HTTP call — a per-endpoint semaphore with a short acquisition deadline, so that surplus work for a saturated destination is rejected quickly and requeued rather than parked in memory holding a worker.

This is bulkhead isolation applied at the destination granularity. Each endpoint gets a fixed allowance of concurrent in-flight deliveries; exceeding it returns immediately with a shed outcome that the queue layer interprets as “retry later, do not count as a failure”. The practical tuning of those allowances, and why a slow-but-successful endpoint never trips a breaker, is the subject of handling slow webhook consumers.

How timeout values decide fleet throughput

Timeouts are usually discussed as a correctness control, but they are a capacity control first. Little’s law states that the concurrency required to sustain a throughput is the throughput multiplied by the mean service time, and in a dispatcher the service time is dominated by whatever the timeouts allow. That makes the relationship between a timeout and the size of your fleet direct and computable.

Start from a healthy fleet: 32 workers per process, 8 processes, 256 concurrent slots, and a mean attempt duration of 250 ms. Capacity is 256 / 0.25, or 1,024 deliveries per second. Now let 2% of traffic go to an endpoint that stopped responding. With a 30-second read timeout, each of those attempts occupies a slot for 30 seconds instead of 250 ms — 120 times longer. At 1,024 deliveries per second, 20 per second are hitting the dead endpoint, and each holds a slot for 30 seconds, so the steady-state occupancy of that one endpoint is 20 × 30 = 600 slots. You have 256. The pool is fully consumed by 2% of the traffic, every other tenant stops being served, and the fleet-wide symptom is total delivery failure caused by a single customer’s outage.

Two levers change that answer and they are multiplicative. Cutting the read timeout to 5 seconds reduces the demand to 100 slots — survivable, though every other tenant now runs at 60% capacity. Adding a per-endpoint concurrency cap of 6 bounds it absolutely: at most 6 slots are ever occupied by that endpoint, and the surplus is shed in milliseconds and requeued. This is the quantitative case for the bulkhead pattern, and it is worth computing for your own numbers before an incident, because the calculation takes two minutes and the conclusion — that a generous timeout is a fleet-wide single point of failure — is not obvious from reading the config.

The same arithmetic gives you the right size for the worker pool in the first place. If your target is 1,000 deliveries per second at a 250 ms mean, you need 250 concurrent slots for the steady state, plus headroom for the tail: budget for the p99 attempt duration on the fraction of traffic that hits it, which in practice means adding 30–50% rather than 10%. Undersizing shows up as queue latency with idle CPU; oversizing shows up as connection-pool contention and memory pressure long before it shows up as anything useful.

Where timeouts end and retries and circuit breakers begin

These three controls are routinely confused because they all “handle failure”, but they observe different signals and catch disjoint failure sets. Timeouts bound a single attempt. Exponential backoff spaces repeated attempts. Circuit breakers suppress attempts to a destination that is statistically dead. Concurrency caps bound how much of your capacity a destination may occupy at once.

The third row is worth internalising. An endpoint returning 200 OK after 25 seconds produces a 0% error rate. A breaker configured on error rate will stay closed forever while that endpoint quietly consumes your entire fleet. Only latency-aware admission control catches it.

TLS verification, redirects, and egress control in the client

The client layer is also a security boundary, and every knob you loosen for reliability has a security consequence. Three defaults deserve explicit configuration rather than inheritance.

Certificate verification stays on, always. The single most common “fix” for a consumer whose certificate chain is broken is verify=False, which converts your dispatcher into an unauthenticated payload broadcaster. Signed payloads reduce the damage, but a disabled verification path is still an exfiltration channel for whatever is in the request body. Reject the endpoint at registration time instead; see TLS configuration for webhook endpoints.

Redirects are disabled by default. follow_redirects=True lets a consumer point a 302 at 169.254.169.254 or an internal address, and your client will faithfully re-issue the signed payload there. If you must support redirects, re-run destination validation on every hop. The full threat model is in preventing SSRF in outbound webhook delivery.

Connection reuse must not outlive destination validation. A long-lived pooled connection pins an IP address that was validated at connect time. If the consumer’s DNS later changes — or was deliberately rebound — you keep shipping to the stale peer. Bound keepalive_expiry so revalidation happens on a known cadence, and drop pooled connections for an endpoint whenever its registration record changes.

A production dispatcher HTTP client

The following is a complete httpx-based client that applies every control described above: layered timeouts, global and per-host connection caps, a per-endpoint admission gate with a shedding deadline, and outcome classification granular enough for the retry layer to make correct decisions.

import asyncio
import time
from dataclasses import dataclass
from typing import Any

import httpx


@dataclass(frozen=True)
class ClientPolicy:
    connect_timeout: float = 3.0
    write_timeout: float = 5.0
    read_timeout: float = 10.0
    pool_timeout: float = 2.0
    max_connections: int = 200
    max_per_host: int = 8
    keepalive_expiry: float = 30.0
    per_endpoint_concurrency: int = 6
    admission_timeout: float = 1.0


class DispatchClient:
    """One instance per dispatcher process. Never construct per request."""

    def __init__(self, policy: ClientPolicy | None = None) -> None:
        self.policy = policy or ClientPolicy()
        self._client = httpx.AsyncClient(
            timeout=httpx.Timeout(
                connect=self.policy.connect_timeout,
                write=self.policy.write_timeout,
                read=self.policy.read_timeout,
                pool=self.policy.pool_timeout,
            ),
            limits=httpx.Limits(
                max_connections=self.policy.max_connections,
                max_keepalive_connections=self.policy.max_connections // 2,
                keepalive_expiry=self.policy.keepalive_expiry,
            ),
            follow_redirects=False,
            verify=True,
        )
        self._gates: dict[str, asyncio.Semaphore] = {}

    def _gate(self, host: str) -> asyncio.Semaphore:
        gate = self._gates.get(host)
        if gate is None:
            gate = asyncio.Semaphore(self.policy.per_endpoint_concurrency)
            self._gates[host] = gate
        return gate

    async def deliver(
        self, url: str, payload: dict[str, Any], headers: dict[str, str]
    ) -> dict[str, Any]:
        host = httpx.URL(url).host
        gate = self._gate(host)
        try:
            await asyncio.wait_for(
                gate.acquire(), timeout=self.policy.admission_timeout
            )
        except asyncio.TimeoutError:
            # Endpoint is saturated. Requeue without recording a failure.
            return {"outcome": "shed", "host": host, "elapsed": 0.0}

        started = time.monotonic()
        try:
            response = await self._client.post(url, json=payload, headers=headers)
            return {
                "outcome": "response",
                "host": host,
                "status": response.status_code,
                "elapsed": time.monotonic() - started,
            }
        except httpx.PoolTimeout:
            return {"outcome": "pool_timeout", "host": host,
                    "elapsed": time.monotonic() - started}
        except httpx.ConnectTimeout:
            return {"outcome": "connect_timeout", "host": host,
                    "elapsed": time.monotonic() - started}
        except httpx.WriteTimeout:
            return {"outcome": "write_timeout", "host": host,
                    "elapsed": time.monotonic() - started}
        except httpx.ReadTimeout:
            # Ambiguous: the consumer may have committed the side effect.
            return {"outcome": "read_timeout", "host": host,
                    "elapsed": time.monotonic() - started}
        except httpx.HTTPError as exc:
            return {"outcome": "transport_error", "host": host,
                    "error": type(exc).__name__,
                    "elapsed": time.monotonic() - started}
        finally:
            gate.release()

    async def aclose(self) -> None:
        await self._client.aclose()

Two details are load-bearing. The client is constructed once per process — a per-request AsyncClient discards the pool and reintroduces a handshake on every delivery. And finally: gate.release() is unconditional, because a cancelled task that never releases its semaphore slot permanently shrinks that endpoint’s allowance until the process restarts.

Failure modes in the client layer

Operating the client layer: metrics, shedding, and CI

The client emits the signals nothing else in the stack can. Instrument at minimum: delivery_phase_duration_seconds as a histogram labelled by phase and outcome; pool_connections_in_use and pool_connections_idle as gauges; connection_reuse_ratio as a rate; and deliveries_shed_total labelled by host. The reuse ratio is the single best early warning — a drop from 0.95 to 0.4 means something started forcing new connections (a peer shortening its idle timeout, a DNS change, or a code path constructing clients per request) and you will see it in latency before you see it in errors.

Alert on rates per host, not fleet aggregates. A dispatcher serving 5,000 endpoints will always have a background hum of connect timeouts from customers whose staging environments are down; the meaningful alert is “endpoint X moved from 0% to 40% connect timeouts in five minutes”, which is exactly the input a circuit breaker consumes.

In CI, timeout behaviour must be tested against a deliberately hostile server, not a mock. Stand up a local endpoint with three routes — one that accepts the connection and sends nothing, one that sends response headers a byte at a time over 60 seconds, and one that closes the socket mid-body — and assert the exact exception type and elapsed time for each. These tests catch the regression that matters most: a library upgrade silently changing which timeout applies to which phase. Deliveries that exhaust every attempt still need somewhere to go, which is dead-letter queue architecture.

Client-layer debugging checklist

• Confirm the client object is a process-level singleton, not constructed per delivery.
• Print the effective timeout config at startup and assert all four phases are non-null.
• Compare connection_reuse_ratio per host against 0.9; investigate any host below it.
• Check keepalive_expiry against the peer’s advertised Keep-Alive: timeout= header where present.
• Verify total attempt budget (connect + write + read) is strictly less than the queue visibility timeout.
• Count pool_timeout outcomes per host — sustained non-zero means the per-host cap is too tight or the endpoint is too slow.
• Reproduce with curl -v --max-time 30 from a dispatcher host to separate consumer latency from egress-path latency.
• Capture ss -tan state time-wait | wc -l on a dispatcher node; a large number indicates connections are not being reused.
• Confirm follow_redirects is disabled and certificate verification is enabled in the running config, not just the source.

Rolling out a client policy change without a fleet-wide surprise

Client policy changes have an unusually wide blast radius for how small the diff looks. Tightening a read timeout from 30 seconds to 8 changes the outcome of every delivery to every slow consumer simultaneously, and those deliveries stop being “slow successes” and start being “read timeouts that get retried” — which means duplicate side effects at consumers who were previously served correctly, just late. Ship it accordingly.

Sequence the rollout in four steps. First, run the new policy in shadow: keep the old timeout in force but record, per delivery, whether the new value would have tripped. A single extra counter labelled by host answers the only question that matters — how many currently-successful deliveries the change would convert into failures — and it costs nothing. Second, act on that list. Any endpoint whose successful deliveries would newly time out gets a per-endpoint override at its measured p99 times three, so the tightened default never applies to a consumer that is legitimately slow. Third, roll the new default to one process in the fleet and compare its outcome distribution against its peers for a full day; a single process is enough because the change is deterministic per delivery, and comparing against peers controls for whatever else is happening that day. Fourth, widen by cohort, holding each stage long enough to cross a nightly batch window.

Rollback needs to be configuration, not a deploy, and it needs to be per endpoint. The realistic failure is not “the new timeout is wrong everywhere” but “the new timeout is wrong for these eleven consumers”, and a global revert to protect eleven endpoints discards the benefit for the other five thousand. Emit the effective policy as a labelled gauge — connect, write, read, pool, per-host cap, admission cap — so an operator can read what a given process is actually running rather than inferring it from a config repository that may be several deploys ahead. Connection settings deserve one extra note: changing max_connections or keepalive_expiry only takes effect for clients constructed after the change, so a rolling restart is part of the rollout rather than an afterthought, and a partially restarted fleet will legitimately show two different reuse ratios until it completes.

Frequently Asked Questions

Related

• Setting connect and read timeouts — deriving real numbers from endpoint latency percentiles.
• Connection pooling and keep-alive for webhook senders — eliminating the per-delivery TLS handshake.
• Handling slow webhook consumers — bulkheads and shedding for endpoints that are slow but successful.
• Circuit breaker patterns — the control that acts on the outcomes this layer produces.
• Resilient delivery & retry strategies — the wider delivery-reliability model this layer sits under.