Connection Pooling and Keep-Alive for Webhook Senders

Your dispatcher’s flame graph is dominated by ssl_do_handshake and your egress NAT gateway is reporting port exhaustion — you are opening a fresh TLS connection for every webhook you send. This page fixes that specific problem, and it belongs to webhook timeout and connection management. Read it alongside setting connect and read timeouts: pooling makes the connect budget almost irrelevant in steady state and critically important during a cold start, so the two settings are chosen together.

The arithmetic is unforgiving. A TLS 1.3 handshake to a public endpoint costs roughly 90–150 ms of wall clock and two round trips, plus CPU for the key exchange, plus an ephemeral source port held in TIME_WAIT for up to 60 seconds after close. At 2,000 deliveries per second, non-pooled delivery burns 120,000 ports per minute against a 28,000-port NAT allocation. You do not run out of bandwidth; you run out of ports, and the symptom looks like random connect failures to healthy endpoints.

Prerequisites

• Python 3.11+ with httpx 0.27+ on httpcore 1.0 (or urllib3 2.x behind requests — the same concepts, different attribute names).
• Shell access to a dispatcher host for ss and curl, or the equivalent container exec.
• The process file-descriptor limit (ulimit -n) and, if you egress through a managed NAT gateway, its per-destination port allocation.
• Delivery volume per destination host over a representative hour — pooling decisions follow the distribution of traffic across hosts, not the total.
• A metrics pipeline that can carry two new gauges and one ratio.

Step 1: Prove you are paying a handshake per delivery

Measure before you tune. Two signals are conclusive: setup time that does not fall to zero on repeat requests, and a large TIME_WAIT population on the dispatcher host.

# 1. Does setup time vanish on the second request to the same host?
curl -s -o /dev/null \
  -w 'run1 tcp=%{time_connect} tls=%{time_appconnect}\n' \
  https://consumer.example/hooks
# Then from inside the dispatcher, compare against the per-phase histogram:
# a healthy pooled fleet reports time_appconnect ~= 0 for >90% of deliveries.

# 2. Is the host churning sockets?
ss -tan state time-wait | wc -l          # thousands => connections are not reused
ss -tan state established '( dport = :443 )' | wc -l   # should be stable, not spiky

A TIME_WAIT count in the thousands on a dispatcher node is the clearest possible evidence: those are connections you opened and closed rather than kept.

Step 2: Hold one client per process and size the global pool

The single most common cause of zero reuse is constructing the client inside the delivery function. Every httpx.AsyncClient(...) (or requests.post(...), which builds a throwaway session internally) creates a fresh pool that is discarded when the call returns.

import asyncio
import os

import httpx

_client: httpx.AsyncClient | None = None
_client_lock = asyncio.Lock()


def _global_pool_size() -> int:
    """Leave headroom for logs, DB pools, and the broker connection."""
    soft_fd_limit = int(os.environ.get("DISPATCH_FD_LIMIT", "4096"))
    return min(400, max(32, (soft_fd_limit - 512) // 4))


async def get_client() -> httpx.AsyncClient:
    global _client
    if _client is not None:
        return _client
    async with _client_lock:
        if _client is None:
            size = _global_pool_size()
            _client = httpx.AsyncClient(
                limits=httpx.Limits(
                    max_connections=size,
                    max_keepalive_connections=size // 2,
                    keepalive_expiry=30.0,
                ),
                timeout=httpx.Timeout(connect=3.0, write=5.0, read=10.0, pool=2.0),
                follow_redirects=False,
            )
    return _client


async def shutdown_client() -> None:
    global _client
    if _client is not None:
        await _client.aclose()
        _client = None

max_connections is a resource guard, not a throughput dial. It must stay comfortably below the process file-descriptor limit because every connection is an FD, and hitting EMFILE takes down unrelated work in the same process — the broker consumer, the metrics exporter, the log writer. max_keepalive_connections is the subset allowed to linger while idle; setting it equal to max_connections means a burst to 400 destinations leaves 400 idle sockets parked for the whole expiry window.

Call shutdown_client() from your process shutdown hook. Sockets abandoned at exit are closed by the kernel with an RST, which the consumer logs as a client error and, if they alert on it, pages someone.

Step 3: Cap connections per host against worker concurrency

A global cap alone leaves the pool first-come-first-served, and the endpoint that holds connections longest wins. This is the noisy-neighbour problem, and it is entirely mechanical: an endpoint answering in 25 s occupies a connection 125 times longer per delivery than one answering in 200 ms.

httpx has no native per-host limit, so enforce it as an admission gate keyed by host — the same mechanism that protects the worker pool:

import asyncio
from collections import defaultdict

MAX_PER_HOST = 8
_host_gates: dict[str, asyncio.Semaphore] = defaultdict(
    lambda: asyncio.Semaphore(MAX_PER_HOST)
)


async def with_host_cap(host: str, coro_factory, *, wait: float = 2.0):
    """Run coro_factory() holding at most MAX_PER_HOST slots for `host`."""
    gate = _host_gates[host]
    try:
        await asyncio.wait_for(gate.acquire(), timeout=wait)
    except asyncio.TimeoutError:
        raise RuntimeError(f"host_pool_saturated:{host}")
    try:
        return await coro_factory()
    finally:
        gate.release()

Choose MAX_PER_HOST from throughput, not intuition: ceil(target_rps_for_host × p95_hold_time). An endpoint you send 20 events per second to, answering at p95 = 250 ms, needs 5 connections; the same rate against a 2-second endpoint needs 40, which is a signal to slow the sender down rather than raise the cap. Worker counts on the other side of the same equation are covered in sizing worker pools for webhook dispatch.

Step 4: Tune keep-alive expiry against peer idle timeouts

Keep-alive is a negotiation you do not control. Your client decides how long to hold an idle connection; the peer decides how long to accept one. When your window is longer than theirs, you will periodically write a request onto a socket the peer has already closed.

Thirty seconds is the default that works nearly everywhere, and it is the value to ship before you have per-host data. Where a consumer advertises Keep-Alive: timeout=N, record it and clamp your expiry to N − 5 for that host. Idle NAT gateways add a second constraint: AWS NAT Gateway silently drops idle flows at 350 seconds, so any client expiry above ~300 s produces connections that appear open on your side and do not exist on theirs.

Step 5: Detect and evict stale pooled connections

A pooled connection has a lifecycle, and the failure state is the one nobody plans for: the peer sent a FIN that your process has not read, so the socket looks reusable until you write to it and receive an RST.

Because the request never reached the application on the other side, exactly one immediate retry on a fresh connection is safe — this is the standard idempotent-write exception, and it is bounded to one attempt so a genuinely broken peer still fails fast:

import httpx

STALE_ERRORS = (httpx.RemoteProtocolError, httpx.ConnectError)


async def post_with_stale_retry(
    client: httpx.AsyncClient, url: str, payload: dict, headers: dict
) -> httpx.Response:
    """Retry exactly once when a pooled socket turns out to be dead."""
    try:
        return await client.post(url, json=payload, headers=headers)
    except STALE_ERRORS:
        # The peer closed the pooled connection; httpx opens a new one here.
        return await client.post(url, json=payload, headers=headers)

Connection reuse must also be invalidated by configuration changes, not only by time. When an endpoint’s URL, certificate pin or registration record changes, drop its pooled connections immediately — otherwise you keep delivering over a connection that was authorised against the previous destination. Slow endpoints deserve the opposite treatment: shorter allowances and tighter admission, as described in handling slow webhook consumers.

Verification and testing

The metric that proves pooling works is the reuse ratio. Derive it from setup time: a delivery whose TLS setup was zero used a pooled connection.

import time

import httpx
from prometheus_client import Counter, Gauge

DELIVERIES = Counter("webhook_deliveries_total", "Deliveries attempted", ["host"])
NEW_CONNS = Counter("webhook_new_connections_total", "Fresh TCP+TLS setups", ["host"])
POOL_IN_USE = Gauge("webhook_pool_connections_in_use", "Leased connections")


async def instrumented_post(
    client: httpx.AsyncClient, host: str, url: str, payload: dict
) -> httpx.Response:
    started = time.monotonic()
    response = await client.post(url, json=payload)
    setup_ms = (time.monotonic() - started) * 1000
    DELIVERIES.labels(host=host).inc()
    # A cold connection cannot complete in under ~40 ms on the public internet.
    if setup_ms > 40 and response.elapsed.total_seconds() * 1000 < setup_ms - 30:
        NEW_CONNS.labels(host=host).inc()
    POOL_IN_USE.set(len(client._transport._pool.connections))
    return response

Then assert reuse directly in an integration test against a local TLS server, counting accepted connections rather than requests:

import asyncio

import httpx
import pytest


@pytest.mark.asyncio
async def test_pool_reuses_one_connection(local_https_server):
    """20 sequential deliveries must open exactly one connection."""
    limits = httpx.Limits(max_connections=10, max_keepalive_connections=10,
                          keepalive_expiry=30.0)
    async with httpx.AsyncClient(limits=limits, verify=False) as client:
        for i in range(20):
            resp = await client.post(local_https_server.url, json={"n": i})
            assert resp.status_code == 200
    assert local_https_server.accept_count == 1


@pytest.mark.asyncio
async def test_expiry_closes_idle_connection(local_https_server):
    limits = httpx.Limits(max_connections=10, max_keepalive_connections=10,
                          keepalive_expiry=1.0)
    async with httpx.AsyncClient(limits=limits, verify=False) as client:
        await client.post(local_https_server.url, json={"n": 1})
        await asyncio.sleep(1.5)
        await client.post(local_https_server.url, json={"n": 2})
    assert local_https_server.accept_count == 2

accept_count == 1 in the first test is the assertion that fails the moment someone reintroduces a per-request client.

Failure modes and gotchas

• A per-request client silently disables everything. async with httpx.AsyncClient() as c: inside the delivery function passes every unit test and destroys pooling in production. The integration test above, asserting accepted-connection counts, is the only reliable guard. Add it to CI, not to a runbook.
• Keep-alive longer than the peer’s idle timeout. This shows up as intermittent RemoteProtocolError or ConnectionResetError at a rate proportional to how idle a host is — busy hosts almost never hit it, quiet hosts hit it constantly. Clamp keepalive_expiry to 30 s and apply the bounded single retry from Step 5.
• Pinned DNS through long-lived connections. A pooled connection keeps using the IP it resolved at creation. If a consumer migrates their endpoint, deliveries keep flowing to the decommissioned address until expiry. This is the strongest argument against multi-minute keep-alive windows, and the reason endpoint config changes must flush the pool.
• HTTP/2 multiplexing hides saturation. Enabling http2=True lets many concurrent requests share one connection, so per-host connection caps stop bounding concurrency at all. If you enable HTTP/2, enforce the limit with the semaphore from Step 3 and treat connection counts as a diagnostic only.

Frequently Asked Questions

Related

• Setting connect and read timeouts
• Handling slow webhook consumers
• Sizing worker pools for webhook dispatch
• Webhook timeout and connection management