Setting Connect and Read Timeouts for Webhook Delivery

You inherited a dispatcher whose HTTP client says timeout=30 and you have been asked to replace that number with something defensible. This is the specific problem this page solves: turning measured endpoint behaviour into four concrete timeout values that hold up in review. It sits under webhook timeout and connection management, and it pairs with connection pooling and keep-alive for webhook senders — pooling changes the connect phase so radically that the two decisions must be made together.

A single scalar is wrong for a structural reason, not a stylistic one. It applies the same patience to a phase whose duration you fully control (TCP connect, which either completes in tens of milliseconds on a working network or never completes at all) and to a phase you do not control at all (the consumer’s business logic). Any value generous enough for the second is catastrophically generous for the first.

Prerequisites

• Python 3.11+ with httpx 0.27+ (requests works too, but exposes only a (connect, read) tuple — no write or pool phase).
• Per-host delivery latency recorded as a histogram, not an average. A prometheus_client.Histogram with buckets at 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 25, 60 seconds is enough.
• At least 24 hours of data per endpoint you intend to tune individually — diurnal traffic makes shorter windows misleading.
• The queue’s visibility timeout (SQS), lock duration (Azure Service Bus) or visibility_timeout (Celery/Redis) written down. Every number below has to fit inside it.
• A per-endpoint configuration store you can write timeout overrides into without a deploy.

Step 1: Measure the endpoint response-time distribution

Do not start from a number you have seen in someone else’s config. Start from the distribution of successful responses per destination, because the timeout’s only job is to sit above the responses you want to keep and below the ones you want to abandon.

The rule that survives contact with production is read timeout ≈ 2 × p99, floored at 5 s and capped at 15 s. The multiplier absorbs normal variance; the floor stops a very fast endpoint from getting a timeout so tight that a single GC pause on their side looks like an outage; the cap stops a chronically slow endpoint from setting policy for your worker pool. When p99 is so high that 2 × p99 exceeds the cap, that is not a tuning problem — it is a capacity problem, and it belongs to handling slow webhook consumers.

Connect timeouts are derived differently, because they measure network topology rather than application behaviour. Take the p99.9 of your successful TCP+TLS setup times across all endpoints — for a dispatcher in one region talking to the public internet this is typically 400–900 ms — and set the connect timeout at roughly 3×, which lands at 2–3 seconds. There is no legitimate reason for it to be higher.

Step 2: Split the budget across connect, write, read and pool

With the numbers in hand, express them as four independent phase budgets rather than one deadline.

import httpx

BASE_TIMEOUT = httpx.Timeout(
    connect=3.0,   # 3x the p99.9 of successful TCP+TLS setup
    write=5.0,     # generous: payloads are small, but peers stall
    read=10.0,     # ~2x endpoint p99, clamped to the 5-15 s band
    pool=2.0,      # shed fast rather than queue on our own pool
)

client = httpx.AsyncClient(timeout=BASE_TIMEOUT, follow_redirects=False)

The pool budget is the one teams forget. It bounds how long a request waits for a free connection from your own client, which is a queue you control. Setting it high converts a local capacity problem into indistinguishable latency; setting it to 1–2 seconds turns it into an explicit, countable shed event.

Step 3: Fit the total attempt deadline inside the retry budget

Phase budgets are not additive in a simple way — a request that spends 3 s connecting and 10 s reading has consumed 13 s, and a slow-drip response can extend the read phase indefinitely if the peer sends a byte every 9 seconds. So add a hard wall-clock deadline on top:

import asyncio
import httpx

ATTEMPT_DEADLINE = 20.0  # seconds, wall clock, per attempt


async def attempt_delivery(
    client: httpx.AsyncClient, url: str, payload: dict, headers: dict
) -> httpx.Response:
    """Bound the whole attempt, not just the individual transport phases."""
    async with asyncio.timeout(ATTEMPT_DEADLINE):
        return await client.post(url, json=payload, headers=headers)

Now check the deadline against two external constraints.

The queue visibility timeout. If your broker re-delivers a message after 60 seconds of invisibility and you make three attempts of up to 20 seconds each with backoff sleeps in between, you will exceed it and a second worker will start delivering the same event while the first is still in flight. The invariant is max_attempts × attempt_deadline + sum(backoff_delays) < visibility_timeout × 0.8.

The delivery SLO. A 20-second deadline with five attempts and exponential backoff puts worst-case delivery latency near four minutes. If your published SLO promises delivery within 60 seconds for 99% of events, the attempt count has to come down or the deadline does.

Picking the attempt count itself is a separate decision, worked through in choosing retry budgets and max attempts.

Step 4: Override the read timeout for known-slow endpoints

A fleet-wide read timeout is a compromise between your fastest and slowest consumer, which means it is wrong for both. Store per-endpoint profiles instead, computed from each endpoint’s own percentiles and refreshed on a schedule:

from dataclasses import dataclass, replace

import httpx


@dataclass(frozen=True)
class TimeoutProfile:
    connect: float = 3.0
    write: float = 5.0
    read: float = 10.0
    pool: float = 2.0

    def to_httpx(self) -> httpx.Timeout:
        return httpx.Timeout(
            connect=self.connect, write=self.write, read=self.read, pool=self.pool
        )


DEFAULT_PROFILE = TimeoutProfile()

# Loaded from config storage, keyed by endpoint id, refreshed every 15 minutes.
OVERRIDES: dict[str, TimeoutProfile] = {
    "ep_slow_erp": replace(DEFAULT_PROFILE, read=25.0),
    "ep_edge_fn": replace(DEFAULT_PROFILE, read=5.0, connect=2.0),
}


def profile_for(endpoint_id: str) -> TimeoutProfile:
    return OVERRIDES.get(endpoint_id, DEFAULT_PROFILE)


async def deliver(
    client: httpx.AsyncClient, endpoint_id: str, url: str, payload: dict
) -> httpx.Response:
    profile = profile_for(endpoint_id)
    return await client.post(url, json=payload, timeout=profile.to_httpx())

Two guardrails keep overrides from becoming a loophole. Clamp every stored value into a permitted band (read between 3 s and 30 s, connect never above 5 s) so a bad config entry cannot reintroduce the 130-second default. And require that any endpoint with an elevated read timeout also carries a reduced concurrency allowance — otherwise you have simply given the slowest consumer permission to hold workers longer.

Step 5: Classify timeout outcomes before retrying

The final step is the one that prevents data corruption. ConnectTimeout and PoolTimeout mean the payload never reached the consumer: retry immediately, they cost nothing. ReadTimeout means the consumer accepted the request and may well have committed the work — your timeout says nothing about whether the side effect happened.

import asyncio

import httpx

RETRY_SAFE = ("connect_timeout", "pool_timeout", "write_timeout")


async def classify_attempt(
    client: httpx.AsyncClient, url: str, payload: dict, headers: dict
) -> dict:
    try:
        async with asyncio.timeout(20.0):
            response = await client.post(url, json=payload, headers=headers)
        return {"outcome": "response", "status": response.status_code}
    except httpx.ConnectTimeout:
        return {"outcome": "connect_timeout", "retry_safe": True}
    except httpx.PoolTimeout:
        return {"outcome": "pool_timeout", "retry_safe": True}
    except httpx.WriteTimeout:
        return {"outcome": "write_timeout", "retry_safe": True}
    except httpx.ReadTimeout:
        # Ambiguous. Retry only with a stable idempotency key.
        return {"outcome": "read_timeout", "retry_safe": False, "ambiguous": True}
    except asyncio.TimeoutError:
        return {"outcome": "deadline_exceeded", "retry_safe": False, "ambiguous": True}

retry_safe=False does not mean “do not retry” — it means “retry only if the request carried a stable idempotency key that the consumer honours”. Every ambiguous outcome must be counted separately in metrics, because a rising read_timeout rate is the leading indicator of a consumer heading toward saturation.

Verification and testing

Assert the exact exception per phase against a deliberately hostile server rather than a mock. A listening socket that never responds exercises the read path; a firewalled address exercises the connect path.

import asyncio
import socket
import time

import httpx
import pytest


@pytest.fixture
def black_hole():
    """A socket that accepts connections and then sends nothing, ever."""
    server = socket.socket()
    server.bind(("127.0.0.1", 0))
    server.listen(8)
    yield f"http://{server.getsockname()[0]}:{server.getsockname()[1]}/"
    server.close()


@pytest.mark.asyncio
async def test_read_timeout_fires_within_budget(black_hole):
    timeout = httpx.Timeout(connect=1.0, write=1.0, read=2.0, pool=1.0)
    async with httpx.AsyncClient(timeout=timeout) as client:
        started = time.monotonic()
        with pytest.raises(httpx.ReadTimeout):
            await client.post(black_hole, json={"id": "evt_1"})
        elapsed = time.monotonic() - started
    assert 1.8 < elapsed < 3.0, f"read timeout drifted: {elapsed:.2f}s"


@pytest.mark.asyncio
async def test_connect_timeout_uses_connect_budget():
    # 10.255.255.1 is non-routable: packets are dropped, not refused.
    timeout = httpx.Timeout(connect=1.0, write=5.0, read=30.0, pool=1.0)
    async with httpx.AsyncClient(timeout=timeout) as client:
        started = time.monotonic()
        with pytest.raises(httpx.ConnectTimeout):
            await client.post("http://10.255.255.1/hooks", json={"id": "evt_2"})
        elapsed = time.monotonic() - started
    # Must fail on the 1 s connect budget, not the 30 s read budget.
    assert elapsed < 2.5, f"connect budget not applied: {elapsed:.2f}s"

The second test is the important one. It fails loudly if a library upgrade or a stray timeout= argument collapses the four phases back into one scalar — the regression that silently reintroduces 30-second connect waits.

For a live check from a dispatcher host, split the phases with curl’s own timers:

curl -s -o /dev/null -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
  --connect-timeout 3 --max-time 20 \
  -X POST -H 'Content-Type: application/json' -d '{"id":"evt_probe"}' \
  https://consumer.example/hooks

If time_appconnect is close to time_connect, the connection was reused and your connect budget is barely exercised in steady state; if ttfb is close to total, the consumer is slow to start responding rather than slow to stream, which is exactly what the read budget governs.

Failure modes and gotchas

• The read timeout is not a total timeout. In most clients the read budget resets on every byte received. A consumer that trickles one byte of headers every 8 seconds will hold a worker indefinitely under a 10-second read timeout. The asyncio.timeout(ATTEMPT_DEADLINE) wrapper from Step 3 is the only thing that bounds this, and it must be present in production, not just in tests.
• Passing a bare number re-collapses the phases. client.post(url, timeout=15) in httpx constructs Timeout(15) — all four phases set to 15 seconds, discarding your carefully derived connect budget. Always pass a Timeout object, and add a lint rule forbidding scalar timeout= arguments in the dispatch package.
• Tuning against averages instead of percentiles. A mean response time of 300 ms is entirely compatible with a p99 of 9 seconds. A timeout set at “5× the average” will abort 1% of legitimate deliveries and manufacture duplicate side effects. Always drive the number from a histogram.
• Connect timeouts too tight for cold pools. Setting connect=0.5 looks decisive until a deploy empties every connection pool at once and a spike of TLS handshakes on a loaded box pushes setup past 500 ms. Keep 2–3 s, and reduce handshake frequency with keep-alive rather than by shaving the budget.

Frequently Asked Questions

Related

• Connection pooling and keep-alive for webhook senders
• Handling slow webhook consumers
• Choosing retry budgets and max attempts
• Webhook timeout and connection management