Tuning Circuit Breaker Thresholds for Webhook Dispatch

You already have a breaker. The code compiles, the states are right, and it still behaves badly in production: it trips when a customer’s endpoint returns two 404s at 3 a.m., and it sits closed for four minutes while a payment provider times out every single request. That is not a design problem, it is a numbers problem. This guide, part of Circuit Breaker Patterns, is about choosing the actual values: the threshold, the window, the minimum throughput, the open duration and the half-open trial count. It assumes the state machine itself already exists — if you are still deciding where breaker state lives, read per-endpoint circuit breaker state machines first and come back to pick the numbers.

A webhook dispatcher makes tuning unusually hard because the traffic distribution per destination is wildly uneven. One endpoint receives 400 requests a second; the one next to it receives three a day. A single global set of thresholds cannot serve both, and a threshold chosen by intuition almost always fires on the quiet endpoint and sleeps through the busy one’s outage.

Prerequisites

• Python 3.10+ with httpx 0.27 for dispatch and pytest 8 for the threshold tests.
• A working Closed/Open/Half-Open breaker with state stored somewhere every dispatch worker can read.
• A delivery log with at least 24 hours of per-attempt records: timestamp, endpoint ID, HTTP status or exception class, and duration. This is the raw material for Step 5; without it you are guessing.
• Per-endpoint request-rate metrics, because minimum throughput is derived from traffic volume rather than picked.
• A retry policy already in place, so the breaker is a second line of defence and not the only one. Pair this with choosing retry budgets and max attempts so the two limits do not fight each other.

Step 1: Choose between failure-count and failure-rate thresholds

There are two families of trip condition. A failure-count breaker trips after N failures (usually N consecutive). A failure-rate breaker trips when the fraction of failures inside a rolling time window crosses a percentage. Count-based breakers are attractive because there is exactly one number to choose, and they are the wrong default for webhooks.

The reason is that a webhook destination rarely fails cleanly. It degrades: 30% of requests time out while 70% succeed. A consecutive-count breaker never sees five failures in a row, so it never trips, and the endpoint bleeds retry budget for hours. Meanwhile the same breaker on a quiet endpoint trips the first time a customer deploys and returns five 502s in a row over ten minutes — a blip you would never have wanted to react to.

Encode the settings as one frozen object so a config is a value you can log, diff and replay:

from dataclasses import dataclass

@dataclass(frozen=True)
class BreakerConfig:
    failure_rate_threshold: float = 0.5   # trip once half the window is failing
    window_seconds: int = 30              # length of the rolling observation window
    min_throughput: int = 20              # observations required before the rate counts
    open_seconds: int = 30                # how long Open lasts before a trial is allowed
    half_open_trials: int = 3             # consecutive successes required to close
    half_open_concurrency: int = 1        # trial requests permitted at the same time

The rolling window itself is a bounded deque of outcomes. Keeping raw events rather than a single counter is what lets you compute a rate and a volume from the same structure:

import time
from collections import deque

class RollingOutcomes:
    """Rolling window of (timestamp, is_failure) observations for one endpoint."""

    def __init__(self, window_seconds: int):
        self.window = window_seconds
        self._events: deque = deque()

    def record(self, is_failure: bool, now: float | None = None) -> None:
        now = time.monotonic() if now is None else now
        self._events.append((now, is_failure))
        self._prune(now)

    def _prune(self, now: float) -> None:
        cutoff = now - self.window
        while self._events and self._events[0][0] < cutoff:
            self._events.popleft()

    def stats(self, now: float | None = None) -> tuple[int, float]:
        """Returns (observations_in_window, failure_rate)."""
        now = time.monotonic() if now is None else now
        self._prune(now)
        total = len(self._events)
        if total == 0:
            return 0, 0.0
        failures = sum(1 for _, failed in self._events if failed)
        return total, failures / total

A 30-second window with a 0.5 rate is a sane starting point for a busy endpoint. Longer windows smooth harder and detect slower; shorter windows detect fast and false-trip more.

Step 2: Set a minimum-throughput guard

This is the single most valuable number in the whole configuration, and the one most often omitted. Without it, a 50% failure-rate threshold trips when an endpoint received exactly two requests in the window and one of them failed. That is not an outage, it is a coin flip, and it will take a customer’s integration offline for the full open duration.

The guard is a floor on the number of observations before the rate is allowed to mean anything:

def should_trip(window: RollingOutcomes, cfg: BreakerConfig,
                now: float | None = None) -> bool:
    total, rate = window.stats(now)
    if total < cfg.min_throughput:
        return False        # not enough evidence; a rate over 2 samples is noise
    return rate >= cfg.failure_rate_threshold

Derive the value from measured traffic rather than picking it. A useful rule is half of the requests you expect to see in one window, floored at a value below which no rate is statistically meaningful:

import math

def min_throughput_for(requests_per_second: float, window_seconds: int,
                       floor: int = 20) -> int:
    """Half the traffic expected in one window, never below `floor`."""
    return max(floor, math.ceil(requests_per_second * window_seconds * 0.5))

An endpoint at 40 rps with a 30-second window gets min_throughput = 600; an endpoint at 0.01 rps gets the floor of 20 and, in practice, will simply never accumulate 20 observations in 30 seconds. That is the correct outcome: a breaker is worthless on an endpoint with no traffic, because failing fast saves nothing when there is nothing to save. For those destinations, rely on the retry cap and on auto-disabling rather than on a breaker.

Recompute these values per traffic class instead of per endpoint. Four buckets cover almost every dispatcher:

Step 3: Decide which HTTP outcomes count as failures

A breaker measures endpoint health. Anything that is not evidence about the endpoint’s health must be excluded from the failure signal, or your breaker becomes a very expensive payload validator.

Timeouts and connection errors are the strongest signal you have — they mean the endpoint could not even accept the request. 5xx responses are almost as strong. A 4xx is the opposite: a 400, 401, 404 or 422 means the endpoint is up, healthy and talking to you, and it is rejecting this specific request. Tripping on those punishes every other event destined for a perfectly working endpoint because one payload was malformed. A 429 is its own category: the endpoint is healthy and explicitly telling you to slow down, which is a backoff instruction, not a trip condition.

Make the classifier a pure function so it is trivially testable and impossible to duplicate accidentally in three places:

import httpx

TRANSPORT_FAILURES = (
    httpx.TimeoutException,
    httpx.ConnectError,
    httpx.ReadError,
    httpx.RemoteProtocolError,
)

def classify(exc: Exception | None, status: int | None) -> str:
    """Returns 'failure', 'success' or 'ignore' for the breaker's rolling window."""
    if exc is not None:
        if isinstance(exc, TRANSPORT_FAILURES):
            return "failure"          # endpoint could not accept the request
        return "ignore"               # our own serialization/config bug
    if status is None:
        return "ignore"
    if 200 <= status < 300:
        return "success"
    if status == 429:
        return "ignore"               # explicit slow-down instruction, not ill health
    if 500 <= status < 600:
        return "failure"
    return "ignore"                   # 4xx: endpoint is up and rejecting this payload

One deliberate exception is worth knowing about: a sustained wall of 401 or 403 responses does mean something is broken, but it is a credential problem, not an availability problem. Route that to endpoint auto-disabling and operator notification rather than to the breaker, so the two failure classes stay distinguishable in your dashboards.

Step 4: Pick the open duration and half-open trial count

The open duration is a bet on how long the endpoint needs to recover. Too short and you hammer a service that is still restarting; too long and you sit dark after it came back. The useful anchor is the endpoint’s observed median recovery time from your delivery log, not a round number. Most webhook consumers recover in tens of seconds (a rolling deploy, a pod restart, a connection pool drain), so 30 seconds is a good default with 15 seconds as an aggressive floor and 120 seconds as the point beyond which you should be dead-lettering rather than waiting.

The half-open trial count is the guard against a premature close. A single successful probe proves almost nothing: it may have hit the one healthy replica behind a load balancer while the other nine are still down. Requiring three consecutive successes before closing raises the bar enough to filter that out while keeping recovery fast. Keep half_open_concurrency at 1 so trials are serialized rather than a burst.

Wiring the four numbers together gives a breaker whose behaviour is fully determined by its config object:

class ThresholdBreaker:
    def __init__(self, cfg: BreakerConfig):
        self.cfg = cfg
        self.window = RollingOutcomes(cfg.window_seconds)
        self.state = "CLOSED"
        self.opened_at = 0.0
        self.trial_successes = 0

    def allow(self, now: float | None = None) -> bool:
        now = time.monotonic() if now is None else now
        if self.state == "OPEN":
            if now - self.opened_at >= self.cfg.open_seconds:
                self.state = "HALF_OPEN"
                self.trial_successes = 0
                return True
            return False
        return True

    def on_result(self, outcome: str, now: float | None = None) -> None:
        now = time.monotonic() if now is None else now
        if outcome == "ignore":
            return                                  # never pollutes the window
        if self.state == "HALF_OPEN":
            if outcome == "success":
                self.trial_successes += 1
                if self.trial_successes >= self.cfg.half_open_trials:
                    self.state = "CLOSED"
                    self.window = RollingOutcomes(self.cfg.window_seconds)
            else:
                self.state = "OPEN"                 # one bad trial re-opens at once
                self.opened_at = now
            return
        self.window.record(outcome == "failure", now)
        if should_trip(self.window, self.cfg, now):
            self.state = "OPEN"
            self.opened_at = now

Note that the window is replaced, not cleared in place, when the breaker closes. Reusing a window that still holds pre-outage failures makes the breaker re-trip on the first post-recovery hiccup.

Step 5: Validate the settings against replayed production traffic

Do not ship thresholds you have only reasoned about. You already have the ground truth in your delivery log: replay it through the breaker offline and score each candidate config on two numbers that trade off against each other — healthy requests wrongly blocked and seconds to detect a real outage.

The harness feeds recorded rows through the same ThresholdBreaker your dispatcher uses, with the recorded timestamp injected instead of the clock:

import csv
import itertools
from dataclasses import asdict

def load_rows(path: str) -> list[tuple[float, str]]:
    """CSV columns: ts (epoch seconds), outcome (success|failure|ignore)."""
    with open(path, newline="") as fh:
        return [(float(r["ts"]), r["outcome"]) for r in csv.DictReader(fh)]

def replay(rows: list[tuple[float, str]], cfg: BreakerConfig) -> dict:
    breaker = ThresholdBreaker(cfg)
    blocked_healthy = 0      # requests that would have succeeded but were refused
    wasted_calls = 0         # failing calls let through after the outage started
    outage_start: float | None = None
    detect_seconds: float | None = None

    for ts, outcome in rows:
        if outcome == "failure" and outage_start is None:
            outage_start = ts
        if not breaker.allow(ts):
            if outcome == "success":
                blocked_healthy += 1
            if detect_seconds is None and outage_start is not None:
                detect_seconds = ts - outage_start
            continue
        if outcome == "failure":
            wasted_calls += 1
        breaker.on_result(outcome, ts)

    return {
        "blocked_healthy": blocked_healthy,
        "wasted_calls": wasted_calls,
        "detect_seconds": detect_seconds,
    }

def sweep(rows: list[tuple[float, str]]) -> list[tuple[dict, dict]]:
    grid = itertools.product([0.3, 0.5, 0.7], [30, 60, 120], [10, 20, 50])
    scored = []
    for rate, window, min_tp in grid:
        cfg = BreakerConfig(failure_rate_threshold=rate,
                            window_seconds=window,
                            min_throughput=min_tp)
        scored.append((asdict(cfg), replay(rows, cfg)))
    # Prefer zero false trips first, then the fastest detection.
    return sorted(scored, key=lambda r: (r[1]["blocked_healthy"],
                                         r[1]["detect_seconds"] or 1e9))

Run the sweep separately against a quiet day (to find false trips) and against a recorded incident (to find detection delay). A config that scores well on only one of the two is not tuned, it is overfitted.

Verification and testing

The four behaviours worth pinning down in tests are the guard, the rate trip, the classifier and the trial count. All of them are deterministic once you inject time:

import pytest

def test_min_throughput_guard_blocks_a_two_request_trip():
    cfg = BreakerConfig(failure_rate_threshold=0.5, window_seconds=30,
                        min_throughput=20)
    b = ThresholdBreaker(cfg)
    b.on_result("failure", now=100.0)
    b.on_result("failure", now=101.0)
    assert b.state == "CLOSED"      # 100% failure rate over 2 samples must not trip

def test_rate_threshold_trips_once_volume_is_reached():
    cfg = BreakerConfig(failure_rate_threshold=0.5, window_seconds=30,
                        min_throughput=20)
    b = ThresholdBreaker(cfg)
    for i in range(20):
        b.on_result("failure" if i % 2 == 0 else "success", now=100.0 + i * 0.1)
    assert b.state == "OPEN"        # 50% over 20 observations is the trip point

def test_client_errors_never_reach_the_window():
    assert classify(None, 404) == "ignore"
    assert classify(None, 429) == "ignore"
    assert classify(None, 503) == "failure"
    assert classify(httpx.ConnectError("refused"), None) == "failure"

def test_half_open_needs_all_three_trials():
    cfg = BreakerConfig(open_seconds=30, half_open_trials=3, min_throughput=1,
                        failure_rate_threshold=0.5)
    b = ThresholdBreaker(cfg)
    b.on_result("failure", now=100.0)
    assert b.state == "OPEN"
    assert b.allow(now=131.0) is True          # open duration elapsed
    b.on_result("success", now=131.0)
    b.on_result("success", now=132.0)
    assert b.state == "HALF_OPEN"              # two successes are not enough
    b.on_result("success", now=133.0)
    assert b.state == "CLOSED"

In production, the signal that your thresholds are wrong is a breaker that opens without a matching rise in endpoint errors. Alert on that divergence directly:

# Breakers that opened while the endpoint's own 5xx rate stayed under 5%.
promtool query instant http://prometheus:9090 \
  'sum by (endpoint) (increase(breaker_open_total[15m]))
     and on (endpoint)
   (rate(webhook_delivery_5xx_total[15m])
      / rate(webhook_delivery_total[15m]) < 0.05)'

# Confirm the guard is doing work: trips that were suppressed for low volume.
grep -c 'breaker_trip_suppressed reason=min_throughput' /var/log/dispatcher.log

A healthy configuration shows a non-zero suppression count (the guard is earning its place), zero opens without a corresponding error-rate rise, and a median time-in-open close to open_seconds rather than several multiples of it.

Failure modes and gotchas

• A failure rate with no minimum-throughput guard. This is the defect that produces most “the breaker opened for no reason” tickets. Two requests, one failure, 50% rate, trip. Every rate-based threshold must be paired with min_throughput, and the guard must be evaluated before the rate as in Step 2 — checking it afterwards leaves a race where a partially filled window still trips.
• Counting 4xx as failures. A customer ships a bad payload, you return 422 to yourself a thousand times, and the breaker takes down delivery for every other event to a completely healthy endpoint. Keep the classifier from Step 3 as the only place status codes are interpreted, and assert on it in tests so nobody “fixes” a flaky integration by widening the failure set.
• Half-open trials that run concurrently. Setting half_open_trials = 3 without setting half_open_concurrency = 1 means three workers probe simultaneously the instant the open duration elapses, which is a small thundering herd against a service that just came back. Serialize trials, and stagger them with jitter in the retry backoff so probe timing does not align across endpoints.
• One global config for every destination. Thresholds tuned on your highest-volume endpoint are catastrophic on a trickle endpoint and vice versa. Bucket destinations by measured traffic class (the table in Step 2), store the class on the endpoint record, and re-evaluate the buckets monthly as traffic shifts.
• Windows that outlive the outage. A window_seconds of 300 on an endpoint that fails for 60 seconds keeps the failure rate elevated long after recovery, so the breaker re-trips repeatedly. Replace the window on close (Step 4) and keep the window shorter than your typical incident, not longer.

Frequently Asked Questions

Related

• Per-Endpoint Circuit Breaker State Machines
• Choosing Retry Budgets and Max Attempts
• Adding Jitter to Webhook Retry Backoff
• Circuit Breaker Patterns