Instrumenting Webhooks with OpenTelemetry for End-to-End Tracing

When a webhook delivery is slow or fails intermittently, the only reliable way to find the culprit is to follow a single event across the producer and consumer as one trace — and that is exactly what this guide builds, extending the broader patterns in Webhook Observability & Monitoring. We will wrap dispatch and delivery in OpenTelemetry spans, propagate the W3C Trace Context traceparent header from producer to consumer, and attach the span attributes that make the resulting trace actionable. Once spans exist, they become the substrate for the targets in defining SLOs for webhook delivery and the signals routed by alerting on webhook delivery failures.

Prerequisites

• Python 3.10+ with opentelemetry-sdk, opentelemetry-exporter-otlp, opentelemetry-instrumentation-requests, and a web framework (FastAPI shown here).
• A running OpenTelemetry collector or any OTLP-compatible backend (Jaeger, Tempo, Honeycomb) reachable from both services.
• An existing webhook dispatcher that sends HTTP POSTs, ideally backed by an outbox so you can record event creation time.
• Signature verification already in place on the consumer per HMAC signature verification — instrument around it, not instead of it.

Step 1: Configure the Tracer and Exporter

Initialize a tracer provider with an OTLP exporter and, critically, set the global propagator to W3C Trace Context so traceparent is the wire format on both ends.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.propagate import set_global_textmap
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

resource = Resource.create({"service.name": "webhook-dispatcher"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://collector:4317")))
trace.set_tracer_provider(provider)

# Ensure traceparent (W3C) is the propagation format on both producer and consumer.
set_global_textmap(TraceContextTextMapPropagator())

tracer = trace.get_tracer("webhook.dispatch")

Three defaults in that setup deserve a decision rather than acceptance. BatchSpanProcessor is doing the most important work: it queues spans and exports them on a background thread, so a slow collector costs you memory instead of latency. Its counterpart, SimpleSpanProcessor, exports synchronously inside the span’s end() call and will add the full collector round-trip to every single delivery — a 40 ms dispatch becomes a 400 ms dispatch the moment the collector degrades, and throughput collapses fleet-wide for a reason nobody looks for. Use SimpleSpanProcessor in tests and never in production. The batch processor’s own knobs matter at volume: max_queue_size defaults to 2048 and schedule_delay_millis to 5000, which is fine up to a few hundred spans per second per process. Above that, raise max_export_batch_size from 512 before raising the queue, because a large queue drained in small batches simply moves the backlog rather than clearing it.

The Resource is the second decision. service.name alone is enough to make traces searchable, but adding service.version and deployment.environment is what lets you answer “did this start with the deploy?” without correlating by hand, and both are free — resource attributes are stored once per batch, not once per span. Add service.instance.id too if your dispatchers are individually addressable; a failure confined to one replica is otherwise invisible in the trace view.

The third is sampling, and the correct choice here is usually to make none. Configure the SDK with the default parent-based, always-on sampler, export everything to a local collector, and put the sampling policy in the collector where it can see complete traces and be changed without a deploy. SDK-side head sampling forces the decision at the first span, before anyone knows whether the delivery failed — which guarantees you discard the traces you would have wanted. The exception is an extreme-volume dispatcher where the export bandwidth itself is the constraint; there, a low-ratio head sample with a rule that always keeps traces already marked sampled by the parent is a reasonable compromise.

Step 2: Open a Dispatch Span and Inject Context

Wrap each delivery attempt in a span. Use inject to write the active context into the outgoing headers; never hand-format traceparent yourself.

import requests
from opentelemetry.propagate import inject

def deliver(event, endpoint, attempt):
    with tracer.start_as_current_span("webhook.deliver") as span:
        span.set_attribute("webhook.endpoint_id", endpoint["id"])
        span.set_attribute("webhook.event_type", event["type"])
        span.set_attribute("webhook.attempt", attempt)
        span.set_attribute("webhook.payload_bytes", len(event["body"]))

        headers = {"Content-Type": "application/json"}
        inject(headers)  # writes traceparent into headers from the active span

        resp = requests.post(endpoint["url"], data=event["body"], headers=headers, timeout=10)
        span.set_attribute("http.response.status_code", resp.status_code)
        if resp.status_code >= 300:
            span.set_status(trace.Status(trace.StatusCode.ERROR, f"status {resp.status_code}"))
        return resp.status_code

Note what this function does not do: it never puts the endpoint URL, the tenant name, or the event ID into the span name. Span names are a low-cardinality dimension in every tracing backend — they are how traces are grouped, aggregated, and priced — and a span named POST https://acme.example.com/hooks/8821 produces one group per endpoint, which turns the backend’s aggregation views into useless noise and, on several managed products, into a billing incident. Keep the name a constant like webhook.deliver and push all variability into attributes, where high-cardinality values are exactly what the storage engine expects.

The timeout is also load-bearing. timeout=10 in requests is the read timeout applied per socket operation, not a bound on total request duration; a consumer that trickles a byte every nine seconds can hold the connection — and the span — open for minutes. Pass a tuple, timeout=(3.05, 10), to set connect and read timeouts separately, and if you need a genuine wall-clock ceiling, enforce it in the worker rather than in the HTTP client. A span whose duration is bounded by nothing is a span you cannot use for a latency percentile.

Finally, wrap the requests.post call so transport exceptions still close the span with a status. As written, a ConnectionError propagates out of the context manager, which OpenTelemetry does record — the context manager sets the status to error and records the exception on exit — but the http.response.status_code attribute is never set, so those spans are missing from any query that filters on status. Set a sentinel such as webhook.transport_error on the span in an except block before re-raising, or accept that “no status code attribute” is your signal for “never got an answer” and write your queries accordingly. Either is fine; not deciding is what leaves a permanent blind spot.

Step 3: Set Span Attributes That Make Traces Actionable

The attributes above — endpoint_id, event_type, attempt, payload_bytes, and http.response.status_code — are what let you filter traces to “attempt > 1 deliveries to endpoint X that returned 5xx.” Add a span event for each retry decision so the backoff schedule is visible inline. Avoid putting the full payload or any secret on the span; record a payload hash instead. Span durations recorded this way are also the raw material for tracking webhook delivery latency percentiles, so keep the attribute names stable once dashboards depend on them.

Step 4: Extract Context and Start a Consumer Span

On the consumer, extract the context from request headers before starting your handler span. This is the join that makes the consumer span a child of the producer span.

from fastapi import FastAPI, Request
from opentelemetry import trace
from opentelemetry.propagate import extract

app = FastAPI()
tracer = trace.get_tracer("webhook.consume")

@app.post("/webhooks")
async def receive(request: Request):
    ctx = extract(dict(request.headers))  # reads traceparent into a context
    with tracer.start_as_current_span("webhook.handle", context=ctx) as span:
        body = await request.body()
        span.set_attribute("webhook.payload_bytes", len(body))
        # verify_signature(...) then process; span auto-closes on exit
        return {"status": "ok"}

Two things about that handler are easy to get wrong. The first is where the span starts relative to signature verification. Starting it before verification means you produce a span for every request that hits the endpoint, including scanner traffic and forged payloads, which is exactly what you want operationally — a spike in spans with webhook.signature_valid=false is a security signal you would otherwise never see — but it also means an attacker controls your span volume. Rate-limit at the edge and set signature_valid as an attribute rather than dropping the span, so the ratio remains visible.

The second is what happens after the handler returns. Most production consumers acknowledge immediately and process asynchronously, which means the interesting work happens after the span has closed and after the trace context has left scope. Propagate it explicitly: serialize the current context into the queued job (a traceparent string field on the message is sufficient), and have the worker call extract on that field and start its processing span as a child. Without this the trace stops at the ingest span and every question about processing latency becomes unanswerable — and it is a genuinely common outcome, because the instrumentation looks complete right up until you need it.

If the producer is an external party you have no contract with, do not adopt their trace ID as your own. Start a fresh trace for the ingest and attach their context as a span link instead. A caller who sets the sampled flag on every request will otherwise dictate what your backend retains, and a caller who reuses a trace ID they saw elsewhere can inject spans into an unrelated investigation. For traffic between services you both operate, adopting the context directly is correct and gives you the single joined trace that makes the whole exercise worthwhile.

Step 5: Record Errors and Close Spans

Set the span status to error on any non-2xx outcome or exception and call record_exception so the stack trace rides on the span. Because the spans use context managers they close automatically, but never swallow exceptions before recording them — an unrecorded error is an invisible failure.

“Non-2xx is an error” is too blunt a rule for webhook delivery, though, and applying it literally makes error-rate panels useless. A 429 with a Retry-After header is the consumer’s rate limiter working correctly; a 410 Gone is a definitive instruction to stop delivering to a decommissioned endpoint; a 409 from an idempotent consumer usually means the event was already processed. None of those are faults in your dispatcher, and marking them ERROR puts them in the same bucket as a 502 from a crashed application. Classify deliberately and record the classification as an attribute so both views remain available.

from opentelemetry.trace import Status, StatusCode

# Outcomes that are expected protocol behaviour, not delivery faults.
BENIGN = {409, 410, 429}

def finish_span(span, status_code, exc=None):
    """Apply the span status and a delivery classification consistently."""
    if exc is not None:
        span.record_exception(exc)
        span.set_status(Status(StatusCode.ERROR, "transport failure"))
        span.set_attribute("webhook.outcome", "transport_error")
        return

    span.set_attribute("http.response.status_code", status_code)
    if 200 <= status_code < 300:
        span.set_status(Status(StatusCode.OK))
        span.set_attribute("webhook.outcome", "acked")
    elif status_code in BENIGN:
        # Leave status UNSET: not an error, but not a successful delivery either.
        span.set_attribute("webhook.outcome", "declined")
    else:
        span.set_status(Status(StatusCode.ERROR, f"status {status_code}"))
        span.set_attribute("webhook.outcome", "rejected")

Leaving the span status UNSET for the benign cases is intentional and worth understanding. In OpenTelemetry, UNSET means “no opinion”, and most backends treat only ERROR as a fault for error-rate calculations, so declined deliveries stay out of the error rate while remaining fully queryable via webhook.outcome. Setting them to OK would be the other defensible choice, but it makes “successful” span counts disagree with your delivery success metric, and two numbers that should match but do not will cost someone an afternoon.

Linking Retries to the Originating Event Trace

The instrumentation so far produces one span per attempt, but says nothing about how the attempts relate to each other. The ideal is one trace per event with an attempt span per delivery, and for short retry schedules that is achievable: open a root span when the outbox row is written, store the serialized traceparent on the row, and restore it before each attempt. The gaps between child spans are then literally the backoff delays, which makes a misconfigured schedule visible without reading any configuration.

The ideal stops working at a specific, measurable point. Tracing backends assemble traces from spans arriving inside a bounded window and index them in blocks measured in minutes; a root span held open across a 26-hour retry budget will be split across many blocks, and the trace view will show attempts 1 and 2 while quietly omitting 3 and 4. That silent truncation is worse than no join at all, because it looks authoritative. The practical cutoff is your backend’s assembly window: if the entire backoff schedule fits inside a few minutes, keep one trace; beyond that, start a new trace per attempt, attach a Link to the original span context, and put webhook.event_id on every span so the join survives even when the link does not.

from opentelemetry.propagate import extract
from opentelemetry.trace import Link, get_current_span

def origin_context(traceparent: str):
    """Rebuild the originating SpanContext from the traceparent stored on the outbox row."""
    ctx = extract({"traceparent": traceparent})
    span_ctx = get_current_span(ctx).get_span_context()
    return span_ctx if span_ctx.is_valid else None

def deliver_late_attempt(event, endpoint, attempt, origin_ctx):
    """Start a fresh trace for a much-delayed retry, linked to the event's first trace."""
    link = Link(origin_ctx) if origin_ctx is not None else None
    with tracer.start_as_current_span(
        "webhook.deliver",
        links=[link] if link else None,
    ) as span:
        span.set_attribute("webhook.event_id", event["id"])
        span.set_attribute("webhook.attempt", attempt)
        span.set_attribute("webhook.endpoint_id", endpoint["id"])
        return deliver(event, endpoint, attempt)

origin_context is the piece that makes this durable: the traceparent string lives on the outbox row, which survives process restarts and rebalances in a way that an in-memory context never does. The rule to remember is that span links are a hint for humans and a filter for queries; event_id is the identifier that actually guarantees you can reassemble the story, and it costs one attribute.

What Instrumentation Costs on the Hot Path

Instrumentation is production code with a production cost, and it is worth knowing the size of that cost before the first person blames tracing for a latency regression. Creating and ending a span with half a dozen attributes costs on the order of 5–20 microseconds of CPU in the Python SDK — against an HTTP round trip measured in tens of milliseconds, that is well under a tenth of a percent and effectively free. Memory is similarly modest: a queued span with attributes is roughly 1–2 KB, so the default 2,048-span queue is a few megabytes per process.

The costs that actually bite are elsewhere. Export bandwidth is real at volume — 2,000 spans per second at 400 bytes each is about 800 KB/s per dispatcher before compression, which is a noticeable line on a cross-zone network bill and a reason to run the collector as a node-local agent rather than sending directly to a regional endpoint. And the queue is bounded, so a slow collector does not slow you down, it silently drops spans; the SDK exposes a dropped-span counter and almost nobody scrapes it, which is why “our traces got patchy about a month ago” is such a common report. Scrape it, alert on any sustained non-zero value, and treat a rising drop rate as a capacity signal for the collector rather than a tracing curiosity.

One more cost is easy to create by accident: attribute cardinality on the backend. Most tracing products index attributes for search, and an attribute like webhook.payload_sha256 with a unique value per span can multiply index size dramatically. Keep unique-per-span values as attributes if you need them for forensics — that is what traces are for — but check whether your backend lets you mark them unindexed, and never promote such a value to the span name or to a resource attribute.

Verification and Testing

Run both services against a local collector and fire one event, then assert the trace joined correctly. A focused integration test extracts the context the producer would send and confirms the trace ID matches:

from opentelemetry.propagate import inject, extract
from opentelemetry import trace

def test_traceparent_round_trips():
    tracer = trace.get_tracer("test")
    with tracer.start_as_current_span("producer") as producer:
        headers = {}
        inject(headers)
        assert "traceparent" in headers
        producer_trace_id = producer.get_span_context().trace_id

    ctx = extract(headers)
    # The extracted span context carries the producer's trace id.
    span_ctx = trace.get_current_span(ctx).get_span_context()
    assert span_ctx.trace_id == producer_trace_id

You can also verify on the wire with curl and inspect that your handler logs the same trace ID it received:

curl -X POST http://localhost:8000/webhooks \
  -H 'Content-Type: application/json' \
  -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' \
  -d '{"type":"payment.succeeded"}'

If the consumer’s logged trace ID does not match the producer’s, work the fault down in a fixed order rather than guessing — the four terminal causes below account for nearly every broken join in practice.

Failure Modes and Gotchas

• Forgetting to set the W3C propagator. OpenTelemetry’s default propagator may differ across SDK versions; if producer and consumer disagree, the header is written in one format and silently ignored in the other. Call set_global_textmap(TraceContextTextMapPropagator()) on both.
• A proxy or CDN stripping the header. Some edges drop unknown headers. Confirm traceparent survives the full path; if it is stripped, allowlist it at the proxy.
• Spans never exported because the process exits first. With BatchSpanProcessor, short-lived dispatch workers can exit before the batch flushes. Call provider.shutdown() on graceful exit, or use a span processor with a short schedule delay for low-volume workers.
• Re-delivered events looking like duplicate traces. A retried delivery should be a new span on the original trace. Combine this with idempotency in webhooks so duplicate handling is observable rather than silent.
• Variable data in the span name. Naming a span after the target URL or the event ID creates one aggregation group per endpoint, which ruins the backend’s grouping views and, on usage-priced products, the invoice. Keep the name constant and put variability in attributes.
• A read timeout mistaken for a total timeout. timeout=10 in requests bounds each socket read, not the whole request, so a trickling consumer can hold a span open for minutes and skew every latency percentile. Pass a (connect, read) tuple and enforce a wall-clock ceiling in the worker.
• The trace ending at the consumer’s acknowledgement. Consumers that ack fast and process asynchronously lose the context at the queue boundary, so processing time becomes untraceable while the instrumentation still looks complete. Serialize the context into the job and extract it in the worker.
• Dropped spans nobody is watching. A saturated exporter queue discards spans by design and reports it only through a counter almost no one scrapes. Alert on any sustained non-zero drop rate, or your traces will thin out silently for weeks.

Frequently Asked Questions

Related

• Defining SLOs for webhook delivery
• Alerting on webhook delivery failures
• Tracking webhook delivery latency percentiles
• Webhook Observability & Monitoring