Auto-disabling failing webhook endpoints without losing the integration

An endpoint that belongs to a mid-size customer has returned 500 to every delivery for three days. Your dispatcher has attempted the same forty thousand events six times each, your retry queue is 240,000 jobs deep and climbing, your egress bill has a visible bump, and nobody at the customer knows, because the only signal was a Slack alert nobody owns. This is the steady state that every provider reaches without an automatic disable path, and the fix is a small amount of bookkeeping on the health record attached to each subscription in webhook subscription management. The endpoint stops receiving traffic, the owner finds out with a specific error in hand, and the subscription — with its identity, secret, and the filters described in event type filtering and subscription scoping — survives intact for a clean restart.

The mechanism has four parts: a counter that only forward-progresses on failure, a threshold that distinguishes a deploy blip from an abandoned endpoint, a notification that reaches a human, and a re-enable path that does not immediately re-break the endpoint with the flood of everything it missed.

Prerequisites

• Runtime: Python 3.12 with psycopg 3.1 for PostgreSQL access and httpx 0.27 for outbound delivery.
• Worker framework: Celery 5.4 or any queue where a task can run after a bounded retry sequence; the counter is written at terminal outcome, not per attempt.
• Schema: a subscription_health table keyed by subscription id, separate from the subscription row itself, so hot delivery writes do not contend with dispatch reads.
• An event archive with a known retention window — backfill can only replay what you still have.
• Metrics: prometheus_client 0.20 or equivalent, plus a transactional email or in-product notification channel that reaches the endpoint owner rather than the account’s billing address.

Step 1: Record every delivery outcome against the subscription

The counter must be consecutive failures, not a failure rate. A rate over a window sounds more principled and behaves worse: an endpoint delivering 10,000 events an hour at a 4% error rate is healthy and noisy, while an endpoint that has failed its last twelve deliveries in a row is dead regardless of what its 24-hour rate says. Consecutive counting is also cheap — one increment, one reset — and it is self-healing, because a single success wipes the history.

Write the outcome once, when the retry sequence for a delivery has terminated. Incrementing per attempt makes the counter a function of your retry policy rather than of the endpoint’s health, and changing the backoff schedule then silently changes your suspension behaviour.

from __future__ import annotations

import enum
from dataclasses import dataclass

import psycopg
from psycopg.rows import dict_row

POOL_DSN = "postgresql:///webhooks"


class Outcome(enum.StrEnum):
    SUCCESS = "success"
    RETRYABLE = "retryable"      # 5xx, 429, timeout, connection reset
    PERMANENT = "permanent"      # 404, 410, 401, TLS failure
    FATAL = "fatal"              # DNS NXDOMAIN, endpoint no longer resolves


@dataclass(frozen=True)
class DeliveryResult:
    subscription_id: str
    outcome: Outcome
    status_code: int | None
    error: str | None


def classify(status_code: int | None, error: str | None) -> Outcome:
    """Map a terminal delivery attempt onto a health outcome."""
    if status_code is not None and 200 <= status_code < 300:
        return Outcome.SUCCESS
    if error and ("NXDOMAIN" in error or "Name or service not known" in error):
        return Outcome.FATAL
    if status_code in (401, 403, 404, 410):
        return Outcome.PERMANENT
    if status_code is not None and 400 <= status_code < 500 and status_code != 429:
        return Outcome.PERMANENT
    return Outcome.RETRYABLE


def record_outcome(conn: psycopg.Connection, result: DeliveryResult) -> dict:
    """Upsert the health row. Called once per delivery, after retries terminate."""
    increment = 0 if result.outcome is Outcome.SUCCESS else 1
    with conn.cursor(row_factory=dict_row) as cur:
        cur.execute(
            """
            INSERT INTO subscription_health AS h (
                subscription_id, consecutive_failures, first_failure_at,
                last_outcome, last_status_code, last_error, last_attempt_at
            )
            VALUES (%(sid)s, %(inc)s,
                    CASE WHEN %(inc)s = 0 THEN NULL ELSE now() END,
                    %(outcome)s, %(code)s, %(err)s, now())
            ON CONFLICT (subscription_id) DO UPDATE SET
                consecutive_failures = CASE WHEN %(inc)s = 0
                    THEN 0 ELSE h.consecutive_failures + 1 END,
                first_failure_at = CASE
                    WHEN %(inc)s = 0 THEN NULL
                    WHEN h.first_failure_at IS NULL THEN now()
                    ELSE h.first_failure_at END,
                last_success_at = CASE WHEN %(inc)s = 0
                    THEN now() ELSE h.last_success_at END,
                last_outcome = %(outcome)s,
                last_status_code = %(code)s,
                last_error = %(err)s,
                last_attempt_at = now()
            RETURNING h.consecutive_failures, h.first_failure_at, h.last_outcome
            """,
            {
                "sid": result.subscription_id,
                "inc": increment,
                "outcome": str(result.outcome),
                "code": result.status_code,
                "err": (result.error or "")[:500] or None,
            },
        )
        return cur.fetchone()

Engineering Note: first_failure_at is as important as the counter. A low-volume subscription that receives one event a week reaches twenty-five consecutive failures only after six months, and a high-volume one reaches it in ninety seconds. Carrying both the count and the elapsed duration is what lets a single policy behave sensibly across both.

Step 2: Choose thresholds that separate a blip from a dead endpoint

A single threshold is always wrong for someone. The policy that holds up in production is graded, and it requires both a count and a duration before it suspends anything.

RETRYABLE failures — 5xx, 429, timeouts — are the normal case during a deploy or an incident, and they deserve patience: suspend only after roughly 25 consecutive failures and at least 12 hours of continuous failure. PERMANENT failures are different in kind. A 404 or 410 means the route is gone; five in a row over an hour is enough. FATAL failures, where the hostname no longer resolves at all, mean the endpoint has been decommissioned and there is nothing to wait for.

That last row is the important one. Consecutive-failure counting deliberately does not catch a flapping endpoint, because suspending an endpoint that is succeeding half the time would lose real deliveries. Flapping is a rate problem, and it belongs to a per-endpoint circuit breaker that sheds load for seconds to minutes without touching subscription state. The two controls compose: the breaker handles transient degradation, the suspension handles death.

from datetime import datetime, timedelta, timezone

SUSPEND_POLICY: dict[Outcome, tuple[int, timedelta]] = {
    Outcome.RETRYABLE: (25, timedelta(hours=12)),
    Outcome.PERMANENT: (5, timedelta(hours=1)),
    Outcome.FATAL: (3, timedelta(minutes=15)),
}


def should_suspend(health: dict) -> bool:
    """Both the count AND the elapsed duration must be satisfied."""
    outcome = Outcome(health["last_outcome"])
    if outcome is Outcome.SUCCESS:
        return False

    limit = SUSPEND_POLICY.get(outcome)
    if limit is None:
        return False
    max_failures, min_age = limit

    if health["consecutive_failures"] < max_failures:
        return False

    first = health["first_failure_at"]
    if first is None:
        return False
    return datetime.now(timezone.utc) - first >= min_age

Engineering Note: Read the thresholds from configuration, not from constants baked into the worker image. The first real incident you hit will be a large customer’s endpoint being suspended during their planned maintenance, and you will want to widen the retryable threshold within minutes rather than within a deploy cycle.

Step 3: Suspend the subscription and notify the owner

Suspension is two operations that must both happen: a guarded status transition, and a notification that reaches a person. Providers routinely get the first right and the second wrong, and a silent suspension is worse than no suspension at all — the customer’s data stops arriving and there is no signal anywhere in their world explaining why.

The notification has to carry the specifics. “Your webhook endpoint was disabled” produces a support ticket; “https://hooks.acme.io/events returned 502 on its last 31 deliveries starting 2026-07-22T04:11Z, most recent error: upstream connect timeout” produces a fix. Include the subscription id, the endpoint URL, the last status code and error body excerpt, the count of buffered events, and a direct link to re-enable.

import logging

from prometheus_client import Counter

log = logging.getLogger(__name__)

suspensions_total = Counter(
    "webhook_subscription_suspensions_total",
    "Subscriptions auto-suspended after sustained delivery failure",
    ["outcome"],
)


def suspend_subscription(conn: psycopg.Connection, subscription_id: str, health: dict) -> bool:
    """Guarded transition: only an active subscription can be suspended."""
    with conn.cursor(row_factory=dict_row) as cur:
        cur.execute(
            """
            UPDATE webhook_subscriptions
               SET status = 'suspended',
                   status_reason = %(reason)s,
                   suspended_at = now()
             WHERE id = %(sid)s
               AND status = 'active'
            RETURNING tenant_id, endpoint_url, owner_email
            """,
            {
                "sid": subscription_id,
                "reason": f"auto_suspend:{health['last_outcome']}:"
                          f"{health['consecutive_failures']}",
            },
        )
        row = cur.fetchone()

    if row is None:
        # Already suspended or revoked by another worker; do not re-notify.
        log.info("suspend_skipped", extra={"subscription_id": subscription_id})
        return False

    conn.execute(
        """
        INSERT INTO subscription_audit (subscription_id, action, payload)
        VALUES (%s, 'auto_suspended', %s)
        """,
        (subscription_id, psycopg.types.json.Json({
            "consecutive_failures": health["consecutive_failures"],
            "last_status_code": health["last_status_code"],
            "first_failure_at": health["first_failure_at"].isoformat(),
        })),
    )
    conn.commit()

    suspensions_total.labels(outcome=health["last_outcome"]).inc()
    notify_owner(
        email=row["owner_email"],
        endpoint_url=row["endpoint_url"],
        subscription_id=subscription_id,
        failures=health["consecutive_failures"],
        status_code=health["last_status_code"],
        error=health["last_error"],
        since=health["first_failure_at"],
    )
    return True


def notify_owner(
    *,
    email: str,
    endpoint_url: str,
    subscription_id: str,
    failures: int,
    status_code: int | None,
    error: str | None,
    since: datetime,
) -> None:
    """Send one actionable message. Deduplicated per suspension, not per attempt."""
    body = (
        f"Webhook endpoint {endpoint_url} has been suspended.\n\n"
        f"Reason: {failures} consecutive failed deliveries since "
        f"{since.isoformat(timespec='seconds')}.\n"
        f"Last response: {status_code or 'no response'} — {error or 'connection failed'}\n\n"
        f"No events are being delivered to this endpoint. Queued events are retained "
        f"for 7 days and can be replayed when you re-enable.\n"
        f"Re-enable: https://dashboard.example.com/webhooks/{subscription_id}"
    )
    send_transactional_email(to=email, subject="Webhook endpoint suspended", body=body)

Engineering Note: Deduplicate the notification on the suspension event, not on a time window. The guarded UPDATE returning None is your idempotency key: if the row was not active, this worker did not cause the suspension and must not send mail. Without that check, a fleet of twelve workers all processing the same terminal outcome sends twelve emails.

Step 4: Re-enable behind a probe and a bounded backfill

Restoring an endpoint by flipping status back to active is how you break it a second time. A subscription suspended for three days has a backlog, and delivering that backlog at full rate to an endpoint that has just come back is a self-inflicted denial of service — often against a consumer who has only just finished the deploy that fixed it.

Re-enable in three stages. First send one probe delivery: a real, signed, clearly-typed endpoint.probe event that the consumer is expected to 200. If it fails, stay suspended and tell the owner immediately, while their fix is still in working memory. If it succeeds, move to active and start a bounded backfill: replay events from suspended_at forward, capped by the archive retention window, at a rate the endpoint has demonstrated it can absorb.

import json
import time
import uuid

import httpx

PROBE_TIMEOUT_S = 10.0
BACKFILL_BATCH = 50
BACKFILL_RATE_PER_SEC = 5


def reenable_subscription(conn: psycopg.Connection, subscription_id: str) -> str:
    with conn.cursor(row_factory=dict_row) as cur:
        cur.execute(
            """
            SELECT id, tenant_id, endpoint_url, secret_ref, suspended_at
              FROM webhook_subscriptions
             WHERE id = %s AND status = 'suspended'
            """,
            (subscription_id,),
        )
        sub = cur.fetchone()
    if sub is None:
        return "not_suspended"

    if not send_probe(sub["endpoint_url"], sub["secret_ref"], subscription_id):
        conn.execute(
            """
            UPDATE webhook_subscriptions
               SET status_reason = 'reenable_probe_failed'
             WHERE id = %s
            """,
            (subscription_id,),
        )
        conn.commit()
        return "probe_failed"

    with conn.cursor() as cur:
        cur.execute(
            """
            UPDATE webhook_subscriptions
               SET status = 'active', status_reason = 'reenabled', suspended_at = NULL
             WHERE id = %s AND status = 'suspended'
            """,
            (subscription_id,),
        )
        if cur.rowcount != 1:
            conn.rollback()
            return "lost_race"

    conn.execute(
        "UPDATE subscription_health SET consecutive_failures = 0, first_failure_at = NULL "
        "WHERE subscription_id = %s",
        (subscription_id,),
    )
    conn.commit()

    backfill_missed_events(conn, sub)
    return "reenabled"


def send_probe(endpoint_url: str, secret_ref: str, subscription_id: str) -> bool:
    payload = {
        "id": f"evt_{uuid.uuid4().hex}",
        "type": "endpoint.probe",
        "subscription_id": subscription_id,
    }
    body = json.dumps(payload).encode()
    headers = sign_delivery(body, secret_ref)
    try:
        response = httpx.post(
            endpoint_url, content=body, headers=headers, timeout=PROBE_TIMEOUT_S
        )
    except httpx.HTTPError:
        return False
    return 200 <= response.status_code < 300


def backfill_missed_events(conn: psycopg.Connection, sub: dict) -> int:
    """Replay archived events from suspension onward, capped by retention and rate."""
    replayed = 0
    cursor_id = ""
    while True:
        with conn.cursor(row_factory=dict_row) as cur:
            cur.execute(
                """
                SELECT id, event_type, payload
                  FROM event_archive
                 WHERE tenant_id = %(tenant)s
                   AND occurred_at >= GREATEST(%(since)s, now() - interval '7 days')
                   AND id > %(cursor)s
                 ORDER BY id
                 LIMIT %(limit)s
                """,
                {
                    "tenant": sub["tenant_id"],
                    "since": sub["suspended_at"],
                    "cursor": cursor_id,
                    "limit": BACKFILL_BATCH,
                },
            )
            batch = cur.fetchall()

        if not batch:
            return replayed

        for event in batch:
            enqueue_delivery(sub["id"], event["id"], event["payload"], priority="backfill")
            replayed += 1
            time.sleep(1 / BACKFILL_RATE_PER_SEC)
        cursor_id = batch[-1]["id"]

Engineering Note: Enqueue the backfill on a separate priority lane. Mixing three days of replay into the live queue delays every current event for every other subscriber on that shard, which converts one customer’s outage into everyone’s latency incident. Where the backlog is large enough to need durable handling and triage, route it through the dead-letter queue architecture instead of holding it in the primary queue.

Verification and testing

• Counter reset: record four failures then one success against a test subscription; assert consecutive_failures is 0 and first_failure_at is NULL. A counter that only resets on a nightly job will suspend healthy endpoints.
• Duration gate: synthesise 30 consecutive retryable failures with first_failure_at set to 10 minutes ago; assert should_suspend() is False, then move it to 13 hours ago and assert True.
• Notification idempotency: run suspend_subscription() concurrently from two connections against the same row; assert exactly one returns True and exactly one email is sent.
• Probe gate: point a suspended subscription at an endpoint returning 503; assert the status remains suspended, status_reason is reenable_probe_failed, and no backfill jobs were enqueued.
• Backfill bound: suspend a subscription 30 days in the past with archived events throughout; assert the replay covers only the retention window and stops there.

# Confirm the graded policy against live data before trusting it.
psql "$DATABASE_URL" -c "
  SELECT s.id, s.endpoint_url, h.consecutive_failures, h.last_status_code,
         now() - h.first_failure_at AS failing_for
    FROM subscription_health h
    JOIN webhook_subscriptions s ON s.id = h.subscription_id
   WHERE s.status = 'active'
     AND h.consecutive_failures >= 25
     AND h.first_failure_at < now() - interval '12 hours'
   ORDER BY h.consecutive_failures DESC;"
# Every row here is a subscription the next outcome will suspend. Eyeball it
# before enabling auto-suspension in production.

Failure modes and gotchas

• Incrementing the counter per attempt instead of per delivery. With six retries per delivery, a threshold of 25 fires after four failed events rather than 25 — and changing the backoff schedule silently changes the suspension policy. Write the health row exactly once, when the retry sequence terminates.
• Suspending on count alone. A subscription that receives one event per day hits 25 consecutive failures roughly a month after it broke, while a busy one hits it in minutes. Requiring both a count and an elapsed first_failure_at age makes one policy work across four orders of magnitude of traffic.
• Suspending silently. If the owner is not told, the outcome is identical to the endpoint being broken, except now you also stopped retrying. Send one specific, actionable message per suspension event, keyed off the guarded UPDATE so a worker fleet does not multiply it, and mirror it into the same channel used for alerting on webhook delivery failures.
• Re-enabling straight into a full backlog. Flipping the status back and releasing three days of queued events at line rate re-breaks the endpoint within seconds, and the second suspension arrives before anyone can react. Probe first, then replay on a separate lane at a bounded rate.
• Treating suspension as deletion. Destroying the row forces the customer through re-onboarding, a new subscription id, a new secret, and the whole registration handshake described in registering and validating webhook endpoints. Keep the row, keep the secret, change the status.

Frequently Asked Questions

Related

• Registering and validating webhook endpoints — the handshake a re-enabled subscription does not have to repeat.
• Event type filtering and subscription scoping — the filter that survives suspension intact.
• Circuit breaker patterns — the seconds-to-minutes control that complements multi-day suspension.
• Webhook subscription management — the subscription state machine suspension belongs to.