Choosing Retry Budgets and Max Attempts for Webhook Delivery

“How many times should we retry?” is the wrong first question. The number that actually matters is the total retry window — how long an event is allowed to stay undelivered before you give up and hand it to a human or a replay job. Once that window is fixed, the attempt count falls out of the backoff curve arithmetically, and the remaining decision is a capacity one: what fraction of the traffic you send to a given endpoint is allowed to be retries. This page sits under Exponential Backoff Algorithms and takes the retry loop built in implementing exponential backoff in Python webhook handlers as given; here we decide the policy constants that loop is configured with.

The failure this prevents is specific and common: a dispatcher configured with max_attempts=10 and a 300-second ceiling keeps hammering a permanently broken endpoint for forty minutes per event. At a thousand failing events, the retry traffic dwarfs the live traffic, the worker pool is fully occupied replaying doomed deliveries, and healthy endpoints starve behind them. A retry budget is what stops a partial outage from consuming the whole dispatcher.

Prerequisites

• Python 3.11+ with asyncio and httpx 0.27 for the dispatch path, plus redis-py 5.x for shared budget counters across dispatcher replicas.
• A working exponential backoff schedule with an explicit ceiling and jitter — the strategies compared in adding jitter to webhook retry backoff change the expected delay per attempt, which changes the arithmetic in Step 2.
• Per-event persistence of attempt and first_attempted_at, so the window can be evaluated across process restarts rather than in-memory only.
• A dead-letter destination that is already wired up and drainable; a retry budget without somewhere to put the losers is just data loss.
• Agreement with the product owner on the delivery contract per event class — this is a business input, not an engineering default.

Step 1: Fix the total retry window before the attempt count

The window is the promise you make to the consumer: “if we accept your event, we will keep trying to deliver it for up to N minutes.” Different event classes deserve different promises. A payment.captured event that drives revenue recognition is worth hours of persistence; a presence.updated event is worthless ninety seconds after it was generated, and retrying it past that point actively harms you by delivering stale state.

Longer windows are not free. Every additional minute widens the duplicate-delivery exposure (a consumer that timed out after successfully processing will see the event again), extends how long queue storage is held, and raises the odds that the payload is semantically stale on arrival. Pick the shortest window that satisfies the contract.

Step 2: Derive max attempts from the backoff curve

With the window fixed, max_attempts is no longer a judgement call. Walk the cumulative delay of your schedule and stop at the last attempt that still starts inside the window. The subtlety is jitter: under full jitter the expected delay is half the exponential ceiling, so a schedule that looks like it spans 62 seconds actually spans about 31 on average. Compute against the expected delay, then sanity-check against the worst case.

from dataclasses import dataclass


@dataclass(frozen=True)
class BackoffPolicy:
    base: float = 2.0        # seconds before the first retry
    factor: float = 2.0      # exponential growth per attempt
    cap: float = 300.0       # per-delay ceiling in seconds

    def ceiling(self, retry_index: int) -> float:
        """Un-jittered delay before retry number `retry_index` (0-indexed)."""
        return min(self.cap, self.base * (self.factor ** retry_index))


def plan_attempts(
    policy: BackoffPolicy,
    window_seconds: float,
    mean_jitter: float = 0.5,   # 0.5 for full jitter, 0.75 for equal jitter, 1.0 for none
    hard_cap: int = 16,
) -> tuple[int, float, list[float]]:
    """Return (max_attempts, expected_span_seconds, cumulative_offsets)."""
    attempts = 1                 # the first delivery fires immediately
    elapsed = 0.0
    offsets = [0.0]
    while attempts < hard_cap:
        nxt = policy.ceiling(attempts - 1) * mean_jitter
        if elapsed + nxt > window_seconds:
            break
        elapsed += nxt
        offsets.append(round(elapsed, 1))
        attempts += 1
    return attempts, elapsed, offsets


if __name__ == "__main__":
    policy = BackoffPolicy(base=2.0, factor=2.0, cap=300.0)
    for label, window in [("notification", 900), ("provisioning", 7200), ("ledger", 86400)]:
        attempts, span, offsets = plan_attempts(policy, window)
        print(f"{label:<14} window={window:>6}s attempts={attempts:>2} "
              f"expected_span={span:>8.1f}s offsets={offsets}")

Running this against a 15-minute notification window yields seven attempts spanning roughly 700 expected seconds; the eighth would start after the window closes, so it is dropped rather than allowed to run past the promise. Two rules keep the result honest. First, always cap the loop (hard_cap) so a misconfigured window cannot produce a fifty-attempt schedule. Second, persist the derived offsets alongside the event: an operator debugging a delivery should be able to see the schedule the event was admitted under, not the schedule the currently deployed config would produce.

Step 3: Cap retries as a share of endpoint capacity

Per-event limits bound one event’s cost. They say nothing about aggregate cost, and aggregate cost is what takes a dispatcher down. If a large consumer starts returning 503 for every request, every in-flight event for that consumer enters its retry schedule simultaneously, and the retry traffic can exceed the original traffic several times over.

The fix is a retry budget: a hard rule that retried requests to an endpoint may not exceed a fixed fraction — 10% is a good starting point — of total attempts to that endpoint over a rolling window. When the endpoint is healthy, retries are rare and the budget is never touched. When the endpoint is broken, the budget is exhausted within seconds and further retries are refused immediately, which converts a retry storm into a clean, fast stream of dead-lettered events.

import time

import redis.asyncio as aioredis


class RetryBudget:
    """Refuses retries once they exceed `ratio` of total attempts per endpoint."""

    def __init__(
        self,
        redis: aioredis.Redis,
        ratio: float = 0.10,
        window_seconds: int = 60,
        min_sample: int = 50,
    ) -> None:
        self.redis = redis
        self.ratio = ratio
        self.window = window_seconds
        self.min_sample = min_sample

    def _keys(self, endpoint_id: str) -> tuple[str, str]:
        bucket = int(time.time()) // self.window
        return (
            f"rb:{endpoint_id}:{bucket}:total",
            f"rb:{endpoint_id}:{bucket}:retry",
        )

    async def record(self, endpoint_id: str, *, is_retry: bool) -> None:
        total_key, retry_key = self._keys(endpoint_id)
        ttl = self.window * 3
        async with self.redis.pipeline(transaction=True) as pipe:
            pipe.incr(total_key)
            pipe.expire(total_key, ttl)
            if is_retry:
                pipe.incr(retry_key)
                pipe.expire(retry_key, ttl)
            await pipe.execute()

    async def allows_retry(self, endpoint_id: str) -> bool:
        total_key, retry_key = self._keys(endpoint_id)
        total_raw, retry_raw = await self.redis.mget(total_key, retry_key)
        total = int(total_raw or 0)
        retries = int(retry_raw or 0)
        if total < self.min_sample:
            return True          # too little signal; do not throttle on noise
        return retries < total * self.ratio

The min_sample guard matters more than it looks. Without it, a low-traffic endpoint that sees two requests and retries one is instantly at a 50% retry ratio and gets locked out of retrying at all — exactly backwards, because a quiet endpoint’s retries cost nothing. Below the sample floor, always allow the retry.

Budget accounting is deliberately global per endpoint rather than per dispatcher replica. Ten replicas each enforcing a local 10% budget enforce nothing collectively; Redis is what makes the fraction mean what it says. This is the same shared-state requirement that drives applying backpressure to webhook consumers toward centralized counters.

Step 4: Separate retryable outcomes from terminal ones

Budget spent on a request that can never succeed is budget stolen from a request that could. A 422 Unprocessable Entity means the consumer parsed your payload and rejected it on its merits — the identical payload will be rejected identically in thirty seconds and in thirty minutes. Retrying it burns an attempt, a slot in the worker pool, and a slice of the endpoint’s retry budget for zero probability of success.

Encode this as data, not as scattered if statements in the dispatch path:

from enum import Enum

import httpx


class Outcome(str, Enum):
    SUCCESS = "success"
    RETRYABLE = "retryable"
    TERMINAL = "terminal"


RETRYABLE_STATUS = frozenset({408, 425, 429, 500, 502, 503, 504, 507, 509})
TERMINAL_STATUS = frozenset({400, 403, 404, 405, 406, 409, 410, 411, 413, 414, 415, 422, 451})

RETRYABLE_EXC = (
    httpx.ConnectTimeout,
    httpx.ReadTimeout,
    httpx.WriteTimeout,
    httpx.PoolTimeout,
    httpx.ConnectError,
    httpx.RemoteProtocolError,
)
TERMINAL_EXC = (
    httpx.UnsupportedProtocol,
    httpx.InvalidURL,
    httpx.TooManyRedirects,
)


def classify_status(status: int) -> Outcome:
    if 200 <= status < 300:
        return Outcome.SUCCESS
    if status in RETRYABLE_STATUS:
        return Outcome.RETRYABLE
    if status in TERMINAL_STATUS:
        return Outcome.TERMINAL
    if 500 <= status < 600:
        return Outcome.RETRYABLE      # unknown 5xx: assume transient
    return Outcome.TERMINAL           # unknown 4xx: assume our fault, do not hammer


def classify_exception(exc: BaseException) -> Outcome:
    if isinstance(exc, RETRYABLE_EXC):
        return Outcome.RETRYABLE
    if isinstance(exc, TERMINAL_EXC):
        return Outcome.TERMINAL
    return Outcome.TERMINAL           # unrecognised failures are never retried blind

The 401/403 row is the one worth arguing about. Treat it as retryable exactly once, and only after force-refreshing the signing material — a rotation that briefly left both sides disagreeing is the realistic cause. Beyond that single retry it is terminal, because repeatedly presenting a rejected credential looks like a credential-stuffing attempt from the consumer’s side. A 410 Gone should not merely be terminal for that event; it is the strongest possible signal for auto-disabling failing webhook endpoints.

Step 5: Stop retrying and dead-letter with a reason

Four independent conditions can end a retry chain, and an operator reading a dead-letter record needs to know which one fired. “Delivery failed” is not an actionable log line; “retry_budget_exhausted after 2 attempts” tells you the endpoint is broadly down, while “max_attempts_exhausted after 8 attempts” tells you this one event is unlucky.

import time
from dataclasses import dataclass


@dataclass
class DeliveryState:
    event_id: str
    endpoint_id: str
    attempt: int                 # 1-based; the number just completed
    first_attempted_at: float    # time.monotonic() of attempt 1


async def stop_reason(
    state: DeliveryState,
    outcome: Outcome,
    budget: RetryBudget,
    *,
    max_attempts: int,
    window_seconds: float,
) -> str | None:
    """Return None to retry, or a machine-readable reason to stop."""
    if outcome is Outcome.SUCCESS:
        return "delivered"
    if outcome is Outcome.TERMINAL:
        return "terminal_response"
    if state.attempt >= max_attempts:
        return "max_attempts_exhausted"
    if time.monotonic() - state.first_attempted_at >= window_seconds:
        return "retry_window_elapsed"
    if not await budget.allows_retry(state.endpoint_id):
        return "retry_budget_exhausted"
    return None


async def finish_delivery(state, outcome, budget, dlq, *, max_attempts, window_seconds):
    reason = await stop_reason(
        state, outcome, budget,
        max_attempts=max_attempts, window_seconds=window_seconds,
    )
    if reason is None:
        await budget.record(state.endpoint_id, is_retry=True)
        return "retry_scheduled"
    if reason != "delivered":
        await dlq.publish({
            "event_id": state.event_id,
            "endpoint_id": state.endpoint_id,
            "attempts": state.attempt,
            "elapsed_seconds": round(time.monotonic() - state.first_attempted_at, 2),
            "stop_reason": reason,
        })
    return reason

Evaluate the gates in this order deliberately: terminal responses first (cheapest and most decisive), then the per-event limits, then the shared budget check, which is the only one that costs a network round trip. Export stop_reason as a Prometheus label on your dead-letter counter — the ratio between retry_budget_exhausted and max_attempts_exhausted is the single most useful signal for telling “one endpoint is down” apart from “delivery is degraded everywhere”. Events that land in the queue with a reason attached are also far easier to triage when you come to replaying events from a dead-letter queue.

Verification and testing

The policy layer is pure logic and deserves real tests — a miscalculated attempt count is invisible until an incident. Test the arithmetic, the budget ratio, and the classification table separately.

import asyncio

import fakeredis.aioredis
import pytest


def test_attempts_fit_inside_the_window():
    policy = BackoffPolicy(base=2.0, factor=2.0, cap=300.0)
    attempts, span, offsets = plan_attempts(policy, window_seconds=900)
    assert span <= 900, "schedule must not exceed the promised window"
    assert offsets[-1] <= 900
    # The next attempt must genuinely not fit, or we stopped too early.
    next_delay = policy.ceiling(attempts - 1) * 0.5
    assert span + next_delay > 900


def test_cap_shortens_deep_schedules():
    capped = BackoffPolicy(base=2.0, factor=2.0, cap=30.0)
    uncapped = BackoffPolicy(base=2.0, factor=2.0, cap=100_000.0)
    assert plan_attempts(capped, 3600)[0] > plan_attempts(uncapped, 3600)[0]


@pytest.mark.asyncio
async def test_budget_blocks_after_the_ratio_is_exceeded():
    redis = fakeredis.aioredis.FakeRedis()
    budget = RetryBudget(redis, ratio=0.10, window_seconds=60, min_sample=50)
    for _ in range(100):
        await budget.record("ep_42", is_retry=False)
    assert await budget.allows_retry("ep_42") is True
    for _ in range(11):
        await budget.record("ep_42", is_retry=True)
    assert await budget.allows_retry("ep_42") is False


@pytest.mark.asyncio
async def test_quiet_endpoints_are_never_throttled():
    redis = fakeredis.aioredis.FakeRedis()
    budget = RetryBudget(redis, ratio=0.10, window_seconds=60, min_sample=50)
    await budget.record("ep_quiet", is_retry=True)
    assert await budget.allows_retry("ep_quiet") is True


def test_terminal_statuses_never_retry():
    for status in (400, 403, 404, 410, 422):
        assert classify_status(status) is Outcome.TERMINAL
    for status in (429, 500, 503, 504):
        assert classify_status(status) is Outcome.RETRYABLE

In staging, confirm the budget end to end by pointing a subscription at an endpoint that returns 503 unconditionally and watching the stop-reason breakdown flip:

# Should show a small number of max_attempts_exhausted, then a flood of
# retry_budget_exhausted once the ratio trips.
curl -s localhost:9090/metrics \
  | grep '^webhook_deadletter_total' \
  | sort -k2 -nr

If every record says max_attempts_exhausted, the budget is not wired into the decision path; if the counter shows retry_budget_exhausted from the very first event, min_sample is too low for that endpoint’s traffic.

Failure modes and gotchas

• Attempt count chosen independently of the ceiling. Raising cap from 60 s to 300 s while leaving max_attempts=8 silently triples the real retry window, so events sit undelivered far past the promise the product made. Whenever you change base, factor, or cap, re-run plan_attempts and let it regenerate max_attempts — never edit the two numbers independently.
• Per-replica budgets that enforce nothing. A 10% retry budget held in a process-local counter becomes an effective 10% per replica; with twelve dispatcher pods the fleet-wide ratio is 120%. Keep the counters in Redis, and alert if the Redis dependency fails open — a budget that silently degrades to “always allow” during a Redis outage is worse than no budget, because it fails exactly when retries are spiking.
• Fixed-window boundaries resetting the budget. The bucketed counters above discard all history at each window rollover, so a broken endpoint gets a fresh 10% allowance every 60 seconds and produces a sawtooth of retry storms. If the sawtooth is visible in your metrics, weight the previous window’s counts into the current ratio (a sliding estimate) rather than shortening the window.
• The window clock restarting on requeue. If first_attempted_at is stamped when a worker picks the event up rather than when the event was first accepted, an event that bounces between workers never ages out and can retry forever. Persist the original timestamp with the event, use a monotonic source for elapsed comparisons, and treat any event whose attempt exceeds max_attempts on load as an immediate dead-letter regardless of its recorded timestamp.

Frequently Asked Questions

Related

• Implementing Exponential Backoff in Python Webhook Handlers — the retry loop these policy constants configure.
• Adding Jitter to Webhook Retry Backoff — why the expected delay is half the ceiling under full jitter.
• Building a Dead-Letter Queue for Failed Webhooks — where every exhausted budget ends up.
• Exponential Backoff Algorithms — the parent overview for backoff design.