Triaging a Dead-Letter Queue That Grew From 40 to 12,000 Overnight

Your dead-letter queue normally holds forty-odd messages: a handful of decommissioned endpoints and a few malformed payloads nobody has cleaned up. This morning it holds twelve thousand, and the graph is still climbing. You are on call, the queue is not the incident but it is the loudest symptom of one, and the worst available move is to hit “redrive all”. This runbook, part of Dead-Letter Queue Architecture, covers the triage: how you should have been alerted, how to collapse twelve thousand messages into the three root causes actually behind them, and how to decide what to do with each group. It assumes the queue itself already exists and is capturing enough context to triage — if the envelopes are thin, fix that first in building a dead-letter queue for failed webhooks.

The mental model that keeps this calm: a dead-letter queue is a sample of your failures, not a backlog to be cleared. Twelve thousand messages is one number. The number that matters is how many distinct failures produced them, and that is almost never more than a handful.

Prerequisites

• Read access to the queue that does not consume messages: SQS long-poll with a short visibility timeout, or a peek API on your broker. Never triage by receiving and deleting.
• Dead-letter envelopes carrying at least endpoint_id, status_code, error_class, attempts and first_failed_at. Triage quality is capped by envelope quality.
• Python 3.10+ with boto3 1.34, plus a scratch PostgreSQL or DuckDB instance to aggregate a sample offline.
• Metrics for queue depth, oldest-message age and enqueue rate, exported at least every 60 seconds.
• A working replay path. This page ends where replaying events from a dead-letter queue begins.

Step 1: Alert on rate of change and queue age

A static depth threshold is the reason you found out at 07:00 instead of 02:10. Pick a threshold low enough to catch this incident and it fires every week on normal churn; pick one high enough to be quiet and it only fires once the damage is hours old. Depth is a level, and levels are the wrong thing to alert on for a queue whose baseline drifts.

Two signals catch this class of incident early and stay quiet the rest of the time. The first is the derivative: how fast the queue is growing, independent of where it started. Forty to twelve thousand overnight is roughly 1,500 messages an hour; a threshold at 200 an hour would have paged within ten minutes of the onset. The second is the age of the oldest message, which catches the opposite failure — a queue that is not growing but is not draining either, because nobody is working it.

Both rules are cheap to express. In Prometheus, with dlq_depth and dlq_oldest_message_age_seconds exported per queue:

# Growth: more than 200 messages added per hour, sustained for 10 minutes.
- alert: DeadLetterQueueGrowing
  expr: sum by (queue) (rate(dlq_enqueued_total[10m])) * 3600 > 200
  for: 10m
  labels: { severity: page }
  annotations:
    summary: "Dead-letter queue is growing faster than 200 messages per hour"

# Stagnation: the oldest message has been sitting for over an hour.
- alert: DeadLetterQueueStale
  expr: max by (queue) (dlq_oldest_message_age_seconds) > 3600
  for: 15m
  labels: { severity: ticket }

Keep a raw-depth alert as well, but demote it to a ticket and set it high. Its only job is to catch a slow leak that never trips the derivative. Wire these into the same channel as your other alerting on webhook delivery failures so the on-call sees queue growth next to the delivery error rate that caused it.

Step 2: Group dead letters by failure signature

Twelve thousand messages is unreadable. Twelve thousand messages grouped by why they failed is usually three lines. The grouping key — the failure signature — is a hash of the fields that describe the failure, deliberately excluding everything that is unique per message.

Include the endpoint identifier (so one broken customer does not hide behind another), the HTTP status code, and the exception or error class. Exclude the payload, the event ID, the timestamp and the correlation ID: those are unique per message and would give you twelve thousand groups of one.

Sample rather than scanning the whole queue. A thousand messages is more than enough to find a cause responsible for most of the growth, and it takes seconds rather than half an hour:

import hashlib
import json
from collections import Counter, defaultdict

import boto3

SIGNATURE_FIELDS = ("endpoint_id", "status_code", "error_class")

def signature(envelope: dict) -> str:
    ctx = envelope.get("failure_context", envelope)
    parts = [str(ctx.get(field, "unknown")) for field in SIGNATURE_FIELDS]
    return hashlib.sha256("|".join(parts).encode()).hexdigest()[:12]

def sample_queue(queue_url: str, sample_size: int = 1000) -> list[dict]:
    """Non-destructive sample: read with a short visibility timeout, never delete."""
    sqs = boto3.client("sqs")
    seen: dict[str, dict] = {}
    while len(seen) < sample_size:
        batch = sqs.receive_message(
            QueueUrl=queue_url,
            MaxNumberOfMessages=10,
            VisibilityTimeout=5,          # returns to the queue almost immediately
            WaitTimeSeconds=2,
            MessageAttributeNames=["All"],
        ).get("Messages", [])
        if not batch:
            break
        for msg in batch:
            seen[msg["MessageId"]] = json.loads(msg["Body"])
    return list(seen.values())

def group_by_signature(envelopes: list[dict]) -> list[tuple[str, int, dict]]:
    counts: Counter = Counter()
    exemplars: dict[str, dict] = {}
    for env in envelopes:
        sig = signature(env)
        counts[sig] += 1
        exemplars.setdefault(sig, env)
    return [(sig, n, exemplars[sig]) for sig, n in counts.most_common()]

def report(queue_url: str) -> None:
    groups = group_by_signature(sample_queue(queue_url))
    total = sum(n for _, n, _ in groups)
    for sig, n, sample in groups[:10]:
        ctx = sample.get("failure_context", sample)
        share = 100 * n / total
        print(f"{sig}  {n:>6}  {share:5.1f}%  "
              f"{ctx.get('endpoint_id')}  {ctx.get('status_code')}  "
              f"{ctx.get('error_class')}")

The output of a real incident looks like this, and it is the whole diagnosis:

Seventy-eight percent of twelve thousand messages is one cause. That is the normal shape of a DLQ spike, and it is why grouping precedes every other decision.

Step 3: Choose replay, discard, or fix-then-replay

Decide per signature group, never per message and never for the queue as a whole. Two questions settle it: is the root cause fixed? and is the event still meaningful if it is applied now?

The second question is the one people skip. A cart.updated event from six hours ago whose cart has since been checked out is not just useless, replaying it can move a customer’s state backwards. Events carrying an absolute state snapshot are usually safe to apply late; events carrying a delta or an instruction usually are not.

Encode the decision so it is recorded rather than remembered:

from dataclasses import dataclass

@dataclass
class Verdict:
    signature: str
    action: str          # "replay" | "discard" | "hold"
    reason: str
    decided_by: str

# Absolute-state events are safe to apply late; instructions and deltas are not.
STALE_UNSAFE_TYPES = {"cart.updated", "inventory.decremented", "session.extended"}

def decide(group: dict, root_cause_fixed: bool, age_seconds: float,
           operator: str, staleness_budget: float = 6 * 3600) -> Verdict:
    ctx = group["exemplar"].get("failure_context", {})
    if not root_cause_fixed:
        return Verdict(group["signature"], "hold",
                       "root cause not confirmed fixed", operator)
    if ctx.get("status_code") == 410:
        return Verdict(group["signature"], "discard",
                       "endpoint permanently gone", operator)
    event_type = group["exemplar"].get("event_type", "")
    if event_type in STALE_UNSAFE_TYPES and age_seconds > staleness_budget:
        return Verdict(group["signature"], "discard",
                       f"stale {event_type} older than staleness budget", operator)
    return Verdict(group["signature"], "replay", "cause fixed, event still valid",
                   operator)

Write every Verdict to durable storage before acting on it. When someone asks in three weeks why 600 events were dropped, the audit record is the only acceptable answer.

Step 4: Drain without triggering a replay storm

Twelve thousand messages released at full worker concurrency is a self-inflicted denial of service against an endpoint that recovered ninety seconds ago. It re-trips the breaker, refills the queue, and turns a one-hour incident into a four-hour one.

Three controls prevent it. Cap the drain rate at a small fraction of the endpoint’s normal throughput. Route every replayed request through the live circuit breaker rather than around it. And abort the run automatically the moment the endpoint’s error rate rises, instead of waiting for a human to notice.

The runner is small, and the important part is that the abort condition is evaluated inside the loop:

import time

class DrainAborted(Exception):
    pass

def drain(messages, deliver, breaker, endpoint_id: str, *,
          rate_per_sec: float = 20.0, error_budget: float = 0.05,
          min_samples: int = 50) -> dict:
    """Replay messages at a fixed rate, aborting if the live error rate rises."""
    interval = 1.0 / rate_per_sec
    sent = failed = skipped = 0
    next_send = time.monotonic()

    for msg in messages:
        if not breaker.allow(endpoint_id):
            raise DrainAborted(f"breaker open for {endpoint_id} after {sent} sent")

        now = time.monotonic()
        if now < next_send:
            time.sleep(next_send - now)
        next_send = max(next_send + interval, time.monotonic())

        try:
            deliver(msg)
            breaker.on_result(endpoint_id, "success")
            sent += 1
        except Exception:
            breaker.on_result(endpoint_id, "failure")
            failed += 1

        total = sent + failed
        if total >= min_samples and failed / total > error_budget:
            raise DrainAborted(
                f"error rate {failed / total:.1%} exceeded budget after {total}")

    return {"sent": sent, "failed": failed, "skipped": skipped}

Start the first run at ten percent of the endpoint’s normal rate against the smallest signature group, not the largest. A small group is a cheap experiment: if it drains clean, raise the rate and move to the 78% group with confidence.

Verification and testing

The two properties worth testing are that grouping actually collapses the queue and that the drain aborts before it does damage:

def test_signature_collapses_identical_failures():
    envelopes = [
        {"failure_context": {"endpoint_id": "ep_7f3a", "status_code": 500,
                             "error_class": "ReadTimeout"},
         "event_id": f"evt-{i}"}                      # unique field, must be ignored
        for i in range(500)
    ]
    envelopes.append({"failure_context": {"endpoint_id": "ep_5c11",
                                          "status_code": 401,
                                          "error_class": "Unauthorized"}})
    groups = group_by_signature(envelopes)
    assert len(groups) == 2
    assert groups[0][1] == 500

def test_drain_aborts_when_error_rate_exceeds_budget():
    class AlwaysAllow:
        def allow(self, _): return True
        def on_result(self, *_): pass

    def deliver(_msg):
        raise RuntimeError("endpoint still broken")

    with pytest.raises(DrainAborted, match="error rate"):
        drain(range(500), deliver, AlwaysAllow(), "ep_7f3a",
              rate_per_sec=1000, min_samples=50)

def test_stale_delta_events_are_discarded_not_replayed():
    group = {"signature": "a1", "exemplar": {"event_type": "cart.updated"}}
    v = decide(group, root_cause_fixed=True, age_seconds=9 * 3600, operator="oncall")
    assert v.action == "discard"

During the live drain, watch three numbers together rather than only queue depth:

# Depth must fall monotonically; a plateau means messages are re-failing.
aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names ApproximateNumberOfMessages ApproximateAgeOfOldestMessage

# The endpoint's error rate must stay flat while the drain runs.
promtool query instant http://prometheus:9090 \
  'rate(webhook_delivery_failed_total{endpoint="ep_7f3a"}[2m])'

# Signature mix must shrink, not shuffle: re-run the grouping mid-drain.
python -m triage report --queue "$DLQ_URL" | head -5

The run is going well when depth falls steadily, the oldest-message age drops rather than climbing, and re-running the grouping shows the dominant signature shrinking instead of being replaced by a new one.

Failure modes and gotchas

• Redriving everything as the first action. The redrive button is right there and it is almost always wrong: it replays messages whose root cause is unfixed, straight back into the queue, while flooding a recovering endpoint. Group first (Step 2), decide per group (Step 3), and drain the smallest group first. If you only remember one rule from this page, it is this one.
• Triage that consumes the queue. Reading with a normal visibility timeout and forgetting to leave messages in flight silently deletes evidence, and a crash mid-triage loses it permanently. Use a short visibility timeout as in Step 2, never call delete_message from a triage script, and copy the sample somewhere durable before you analyse it.
• A signature that includes a unique field. Adding event_id, the payload hash or the timestamp to the signature produces one group per message and makes the whole exercise useless. If your top group has a count of 1, the signature is too specific — drop fields until the distribution has a clear head.
• Ignoring the age of the oldest message. A queue holding steady at 12,000 looks calm on a depth graph while its oldest message quietly ages past your idempotency window, so replaying it later double-applies. Track age as a first-class signal (Step 1) and treat crossing the dedup TTL as a hard deadline for the discard-or-replay decision.
• Draining with no abort condition. A drain loop that only checks the breaker at startup will happily push twelve thousand messages into an endpoint that failed on message fifty. The error-budget check belongs inside the loop, evaluated after every message, as in Step 4.

Frequently Asked Questions

Related

• Building a Dead-Letter Queue for Failed Webhooks
• Replaying Events From a Dead-Letter Queue
• Alerting on Webhook Delivery Failures
• Dead-Letter Queue Architecture