Tracking Webhook Delivery Latency Percentiles from Event Creation to Acknowledgement

A support ticket that says “your webhooks were slow yesterday” is unanswerable until you can point at a percentile over a defined interval, which is the instrumentation this guide builds on top of Webhook Observability & Monitoring. The scenario is a dispatcher that already emits spans from instrumenting webhooks with OpenTelemetry but reports a single average delivery time — a number that stayed at 900 ms all through an incident where the slowest 1% of events took four minutes. We will separate the clocks that actually matter, emit them as Prometheus histograms with deliberately chosen buckets, compute p50/p95/p99 correctly across a fleet of dispatchers, and keep the label set small enough that the time series database survives. The resulting series are exactly what the targets in defining SLOs for webhook delivery are measured against.

Prerequisites

• Python 3.11+ with prometheus-client 0.20 or later, and a dispatcher you can edit on both the enqueue and delivery paths.
• Prometheus 2.51+ (or a compatible store) scraping every dispatcher instance, with recording-rule support.
• An event record that stores a producer-side created_at timestamp — without it, only the tail of the pipeline is measurable.
• Reasonably synchronised clocks (NTP or better) across producers and dispatchers, since three of the four intervals span processes.

Step 1: Record the four clocks that bound delivery latency

Delivery latency is not one duration. Four timestamps bound it — the moment the event was created in the producing transaction, the moment it landed on the delivery queue, the moment the first HTTP attempt began, and the moment a terminal 2xx was recorded — and each gap has a different owner and a different fix. Collapsing them into a single “delivery time” is how a queue that is 90 seconds behind hides behind endpoints that respond in 40 ms.

Capture the timestamps in one small object that the dispatcher threads through the delivery path, and observe each interval as its own histogram.

# delivery_metrics.py
from __future__ import annotations

import time
from dataclasses import dataclass
from prometheus_client import Counter, Histogram

# Buckets straddle the thresholds the SLO cares about: 1s, 15s, 60s, 15m.
E2E_BUCKETS = (0.5, 1, 2, 5, 10, 15, 30, 60, 120, 300, 900, 1800, 3600)
ATTEMPT_BUCKETS = (0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30)

outbox_lag = Histogram(
    "webhook_outbox_lag_seconds",
    "Seconds from event creation to enqueue on the delivery queue.",
    ["event_type"],
    buckets=E2E_BUCKETS,
)
queue_wait = Histogram(
    "webhook_queue_wait_seconds",
    "Seconds from enqueue to the start of the first delivery attempt.",
    ["endpoint_group"],
    buckets=E2E_BUCKETS,
)
attempt_duration = Histogram(
    "webhook_attempt_duration_seconds",
    "Seconds spent inside a single HTTP delivery attempt.",
    ["endpoint_group", "outcome"],
    buckets=ATTEMPT_BUCKETS,
)
delivery_e2e = Histogram(
    "webhook_delivery_seconds",
    "Seconds from event creation to terminal acknowledgement.",
    ["endpoint_group", "event_type"],
    buckets=E2E_BUCKETS,
)
attempts_total = Counter(
    "webhook_attempts_total",
    "Delivery attempts by outcome.",
    ["endpoint_group", "outcome"],
)


@dataclass
class DeliveryClock:
    """Wall-clock timestamps for one event, carried with the delivery job."""

    event_type: str
    endpoint_group: str
    created_at: float          # set by the producer, stored with the event
    enqueued_at: float | None = None
    first_attempt_at: float | None = None

    def mark_enqueued(self) -> None:
        self.enqueued_at = time.time()
        outbox_lag.labels(self.event_type).observe(
            self.enqueued_at - self.created_at
        )

    def start_attempt(self) -> float:
        now = time.time()
        if self.first_attempt_at is None:
            self.first_attempt_at = now
            queue_wait.labels(self.endpoint_group).observe(
                now - (self.enqueued_at or self.created_at)
            )
        return time.monotonic()

    def end_attempt(self, started: float, outcome: str) -> None:
        # Durations use the monotonic clock; NTP steps must not create
        # negative observations.
        attempt_duration.labels(self.endpoint_group, outcome).observe(
            time.monotonic() - started
        )
        attempts_total.labels(self.endpoint_group, outcome).inc()

    def mark_acked(self) -> None:
        delivery_e2e.labels(self.endpoint_group, self.event_type).observe(
            time.time() - self.created_at
        )

Observe delivery_e2e exactly once per event, on the terminal acknowledgement — never per attempt. An event that succeeds on retry 4 contributes one sample of its full lifetime, which is what the integrator actually experienced and what the delivery guarantee level promises.

Step 2: Choose histogram buckets instead of summary quantiles

An average of 900 ms tells you nothing about the tail because it is dominated by the many fast deliveries: nine hundred events at 100 ms and one hundred events at 8 s average to 890 ms, and no consumer ever experienced 890 ms. Percentiles are the only honest summary, and how you compute them is a fork in the road. A Prometheus summary computes quantiles inside each process; a histogram exports bucket counts and lets the server compute quantiles at query time. For a fleet of dispatchers, only the histogram is aggregatable — you can add bucket counts across ten instances, but you cannot average ten p95 values into a fleet p95.

Bucket choice is the one decision you cannot change retroactively, so place boundaries on the numbers you will be asked about. If the SLO says “acknowledged within 15 minutes”, you need an le="900" bucket, because the ratio of that bucket to the total is the SLI. Add a boundary at each threshold you might alert on — 1 s, 15 s, 60 s, 900 s — and accept coarse resolution between them. A missing boundary means the answer to “what fraction landed inside the objective” has to be interpolated, and interpolation across a wide bucket is a guess.

Step 3: Compute p50, p95, and p99 across every instance

The correct incantation aggregates bucket rates by le before applying histogram_quantile. Any query that computes a quantile per instance and then averages is arithmetically meaningless, no matter how reasonable the dashboard looks.

groups:
  - name: webhook_latency
    interval: 30s
    rules:
      # Correct: sum bucket rates by le first, then compute the quantile.
      - record: webhook:delivery_latency_seconds:p50
        expr: |
          histogram_quantile(
            0.50,
            sum by (le) (rate(webhook_delivery_seconds_bucket[5m]))
          )
      - record: webhook:delivery_latency_seconds:p95
        expr: |
          histogram_quantile(
            0.95,
            sum by (le) (rate(webhook_delivery_seconds_bucket[5m]))
          )
      - record: webhook:delivery_latency_seconds:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (le) (rate(webhook_delivery_seconds_bucket[5m]))
          )
      # Per-endpoint-group p95: keep le plus the one label you group by.
      - record: webhook:delivery_latency_seconds:p95_by_group
        expr: |
          histogram_quantile(
            0.95,
            sum by (le, endpoint_group) (rate(webhook_delivery_seconds_bucket[5m]))
          )

Two details decide whether these numbers are trustworthy. The range in rate() must cover at least four scrape intervals, or a restarted instance produces gaps that read as a latency drop. And histogram_quantile returns the upper bound of the bucket containing the quantile, linearly interpolated within it, so a p99 that lands in the top finite bucket is reported as that boundary and a p99 beyond it is +Inf. If p99 pins to your largest boundary, the tail is longer than your buckets can see, not exactly that value.

You can reproduce the server’s arithmetic locally, which is how you convince yourself a suspicious dashboard is right:

# quantile_check.py
from __future__ import annotations

import math


def quantile_from_buckets(
    cumulative: list[tuple[float, float]], q: float
) -> float:
    """Mirror of PromQL histogram_quantile over cumulative bucket counts.

    `cumulative` is [(upper_bound, count_le_bound), ...] sorted ascending,
    ending with (inf, total).
    """
    total = cumulative[-1][1]
    if total == 0:
        return float("nan")

    rank = q * total
    lower_bound = 0.0
    lower_count = 0.0

    for upper, count in cumulative:
        if count >= rank:
            if math.isinf(upper):
                return lower_bound  # tail exceeds the largest finite bucket
            if count == lower_count:
                return upper
            fraction = (rank - lower_count) / (count - lower_count)
            return lower_bound + (upper - lower_bound) * fraction
        lower_bound, lower_count = upper, count

    return cumulative[-2][0]


if __name__ == "__main__":
    observed = [
        (1.0, 900.0),
        (15.0, 980.0),
        (60.0, 995.0),
        (900.0, 999.0),
        (float("inf"), 1000.0),
    ]
    for q in (0.50, 0.95, 0.99):
        print(f"p{int(q * 100)} = {quantile_from_buckets(observed, q):.2f}s")

That sample distribution prints p50 = 0.56s, p95 = 9.75s, and p99 = 45.00s — a mean well under one second sitting on top of a 45-second p99, which is precisely the shape that averages conceal and that drives alerting on webhook delivery failures to fire on burn rate instead.

Step 4: Bound label cardinality on endpoint and tenant

A histogram costs one series per bucket per label combination, and it is trivially easy to build a metric that outnumbers your event volume. With 13 buckets plus +Inf, _sum and _count, a single label pair of 500 endpoints and 2,000 tenants produces millions of active series — enough to evict the rest of your monitoring.

The fix is not to abandon per-endpoint visibility but to bound it. Keep a periodically refreshed allowlist of the endpoints that carry most of the traffic, map everything else to other, and push per-tenant identity into exemplars and structured logs where high cardinality is cheap.

# label_policy.py
from __future__ import annotations

import threading
from collections import Counter as FreqCounter

_TOP_N = 20
_lock = threading.Lock()
_allowlist: frozenset[str] = frozenset()
_recent = FreqCounter()


def endpoint_group(endpoint_id: str) -> str:
    """Collapse endpoint ids to a bounded label value."""
    with _lock:
        _recent[endpoint_id] += 1
        return endpoint_id if endpoint_id in _allowlist else "other"


def refresh_allowlist() -> frozenset[str]:
    """Promote the busiest endpoints; call on a timer, e.g. every 10 minutes."""
    global _allowlist
    with _lock:
        top = {name for name, _ in _recent.most_common(_TOP_N)}
        _recent.clear()
        _allowlist = frozenset(top)
        return _allowlist


def observe_with_exemplar(histogram, seconds: float, trace_id: str, tenant: str):
    """Attach the identifying detail as an exemplar, not as a label."""
    histogram.observe(seconds, exemplar={"trace_id": trace_id, "tenant": tenant})

Exemplars keep the drill-down path intact: the p99 bar on a dashboard still links to a specific trace, but the tenant identity lives on one sample rather than on a whole series. Anything you truly need to slice by — event type, or a coarse endpoint group — must stay a small, enumerable set, the same discipline applied to queue partitioning in webhooks vs polling vs WebSockets when consumer counts grow.

Verification and testing

Assert the histogram behaviour against an isolated registry so a refactor cannot silently move an observation to the wrong metric or drop a bucket boundary.

# test_delivery_metrics.py — pytest -q
import time
from prometheus_client import CollectorRegistry, Histogram

E2E_BUCKETS = (0.5, 1, 2, 5, 10, 15, 30, 60, 120, 300, 900, 1800, 3600)


def _bucket(registry, name, le, **labels):
    return registry.get_sample_value(
        f"{name}_bucket", {**labels, "le": str(le)}
    )


def test_slo_boundary_bucket_exists_and_counts():
    registry = CollectorRegistry()
    h = Histogram(
        "webhook_delivery_seconds",
        "test",
        ["endpoint_group"],
        buckets=E2E_BUCKETS,
        registry=registry,
    )

    for seconds in (0.4, 3.0, 12.0, 240.0, 1500.0):
        h.labels("other").observe(seconds)

    # The 900s SLO boundary must exist, or the SLI cannot be computed.
    assert _bucket(registry, "webhook_delivery_seconds", "900.0",
                   endpoint_group="other") == 4.0
    assert registry.get_sample_value(
        "webhook_delivery_seconds_count", {"endpoint_group": "other"}
    ) == 5.0


def test_end_to_end_observed_once_per_event():
    registry = CollectorRegistry()
    h = Histogram(
        "webhook_delivery_seconds", "test", buckets=E2E_BUCKETS,
        registry=registry,
    )
    created_at = time.time() - 42.0
    # Three attempts, one terminal acknowledgement.
    h.observe(time.time() - created_at)
    assert registry.get_sample_value("webhook_delivery_seconds_count") == 1.0

Then check the live series on the wire. curl -s localhost:9100/metrics | grep webhook_delivery_seconds_bucket | wc -l should return a number in the low hundreds, not the tens of thousands — that single count is the fastest cardinality regression test you have. Finally, run promtool query instant against the recording rules and confirm p95 tracks a synthetic event you timed by hand, and that count(count by (endpoint_group) (webhook_delivery_seconds_count)) stays at or below your allowlist size plus one.

Failure modes and gotchas

• Averaging percentiles across instances. avg(histogram_quantile(0.95, rate(...))) and any dashboard built on per-instance quantiles produce a number with no statistical meaning; during a partial outage it reads lower than the truth. Always sum by (le) first, then take the quantile.
• Starting the clock at the first HTTP attempt. Outbox lag and queue wait are where multi-minute delays actually accumulate, and an attempt-scoped timer reports them as zero. Persist created_at with the event and measure the full span, observing it once at terminal acknowledgement.
• Buckets that miss the SLO threshold. If the objective is 15 minutes and the nearest boundaries are 300 s and 1800 s, every compliance number is an interpolation across a 25-minute-wide bucket. Put an explicit boundary on every threshold you alert or report on before you publish a target.
• Percentiles over sparse traffic. A low-volume endpoint group may see a handful of events per scrape window, so p99 is computed from two samples and swings wildly. Widen the rate() range for those groups, alert on absolute latency instead, and lean on retry-exhaustion counters and exponential backoff metrics for the real signal.

Frequently Asked Questions

Related

• Defining SLOs for webhook delivery
• Instrumenting webhooks with OpenTelemetry
• Alerting on webhook delivery failures
• Webhook Observability & Monitoring