Benchmarking Webhook Throughput with k6

“Our webhook endpoint handles 2,000 requests per second” is almost always a claim about a load generator, not about a receiver. The number that matters operationally is narrower: how many signature-verified, enqueued, acknowledged deliveries per second the receiver sustains before its handler latency departs from flat — and, separately, how many its workers can actually drain. This guide builds that measurement on top of load testing webhook endpoints using a stepped k6 benchmark, and it is the companion to simulating webhook traffic spikes: a spike test asks “does it survive a sudden herd”, a benchmark asks “where exactly is the ceiling, and which component owns it”.

The distinction that trips people up is admission versus completion. A receiver that returns 202 Accepted in four milliseconds and pushes to a queue will happily admit 5,000 requests per second while its workers drain 400. Every HTTP metric says the endpoint is healthy; the backlog says the system is minutes from collapse. A benchmark that measures only http_req_duration will report the wrong number by an order of magnitude, so the first step is not writing the k6 script — it is making the receiver tell you what is happening behind the 202.

Prerequisites

• k6 0.50+ installed locally or as the grafana/k6 container image; the built-in k6/crypto and k6/metrics modules are all this benchmark needs.
• A staging receiver on production-shaped hardware, with the same worker count, connection pool sizes, and database as production. Benchmarking a laptop tells you about the laptop.
• The real signing scheme and a staging secret exported as WEBHOOK_SECRET, because signature verification is often 20–40% of per-request CPU and skipping it inflates the result.
• Python 3.11+ on the receiver side for the instrumentation middleware, and for the analysis script that reads k6’s JSON summaries.
• A load generator host with raised file-descriptor limits (ulimit -n 65535) and enough CPU headroom that it is not the thing you end up measuring.

Step 1: Expose the four numbers the benchmark depends on

Four quantities describe a webhook receiver under load, and only two of them are visible to k6 by default. Offered rate (what you asked for) and achieved rate (what actually left) come free from the arrival-rate executor. Handler duration and queue depth live inside the receiver, so send them back on the response where the load generator can record them per request.

A small FastAPI middleware is enough on the receiver side:

import time

from fastapi import FastAPI, Request

app = FastAPI()
queue: "asyncio.Queue" = ...          # the receiver's existing bounded work queue


@app.middleware("http")
async def expose_benchmark_signals(request: Request, call_next):
    started = time.perf_counter()
    response = await call_next(request)
    elapsed_ms = (time.perf_counter() - started) * 1000
    # Handler-only cost: excludes client network time and k6's own overhead.
    response.headers["X-Handler-Duration-Ms"] = f"{elapsed_ms:.3f}"
    response.headers["X-Queue-Depth"] = str(queue.qsize())
    return response


@app.get("/benchmark/health")
async def health() -> dict[str, int | float]:
    return {"queue_depth": queue.qsize(), "queue_max": queue.maxsize}

Emitting these as response headers rather than scraping Prometheus separately is deliberate: k6 records them against the exact request that observed them, so a p95 handler duration and the queue depth at that moment end up in the same summary, aligned in time, with no clock-correlation work afterwards.

Step 2: Generate a valid signature for every request

An unsigned benchmark measures a code path production never runs. Reproduce the signing scheme exactly, including the canonical string the receiver reconstructs — a mismatch here means you measure the rejection path, which is much cheaper and will flatter your numbers enormously.

// bench.js
import http from "k6/http";
import crypto from "k6/crypto";
import { check } from "k6";
import { Trend, Gauge, Counter } from "k6/metrics";

const SECRET = __ENV.WEBHOOK_SECRET;
const HOST = __ENV.HOST;
const PATH = __ENV.PATH_ || "/webhooks/orders";

export const handlerMs = new Trend("handler_duration_ms", true);
export const queueDepth = new Gauge("receiver_queue_depth");
export const accepted = new Counter("accepted_events");

function signedRequest() {
  const ts = Math.floor(Date.now() / 1000).toString();
  const body = JSON.stringify({
    id: `evt_${__VU}_${__ITER}_${ts}`,   // unique: defeats dedup and caching
    type: "order.created.v1",
    created_at: ts,
    data: { amount: 1000 + (__ITER % 9000), currency: "USD" },
  });
  // Canonical string must match the receiver byte for byte.
  const mac = crypto.hmac("sha256", SECRET, `${ts}.${body}`, "base64");
  return {
    body,
    headers: {
      "Content-Type": "application/json",
      "X-Webhook-Timestamp": ts,
      "X-Webhook-Signature": `v1=${mac}`,
    },
  };
}

export default function () {
  const { body, headers } = signedRequest();
  const res = http.post(`${HOST}${PATH}`, body, { headers });

  const handler = res.headers["X-Handler-Duration-Ms"];
  if (handler !== undefined) {
    handlerMs.add(parseFloat(handler));
  }
  const depth = res.headers["X-Queue-Depth"];
  if (depth !== undefined) {
    queueDepth.add(parseInt(depth, 10));
  }
  const ok = check(res, {
    "accepted": (r) => r.status === 202 || r.status === 200,
    "signature accepted": (r) => r.status !== 401,
  });
  if (ok) {
    accepted.add(1);
  }
}

Two properties are worth guarding. The event ID varies per iteration, so consumer-side deduplication and any HTTP cache in front of the receiver cannot short-circuit the work — a fixed payload can make a receiver look twice as fast as it is. And the signature accepted check exists to catch the silent failure where every request 401s and the benchmark cheerfully reports 8,000 req/s of rejections.

Step 3: Step the arrival rate and hold each level

Use an open-model executor. ramping-arrival-rate offers a fixed number of iterations per second regardless of how slow responses become; a closed model (constant-vus) throttles itself as latency rises and will report a plateau that is really just your VU count divided by response time.

Hold each step long enough for queues to reach steady state. Sixty seconds is a reasonable floor; if your worker pool has a warm-up (JIT, connection pools, a cold page cache), the first step should be discarded entirely.

const STEP = Number(__ENV.STEP || 200);   // rps increment per level
const HOLD = __ENV.HOLD || "60s";
const RAMP = "10s";

export const options = {
  discardResponseBodies: false,   // needed: we read response headers
  scenarios: {
    staircase: {
      executor: "ramping-arrival-rate",
      startRate: STEP,
      timeUnit: "1s",
      preAllocatedVUs: 800,
      maxVUs: 4000,
      stages: [
        { target: STEP * 1, duration: HOLD },
        { target: STEP * 2, duration: RAMP }, { target: STEP * 2, duration: HOLD },
        { target: STEP * 3, duration: RAMP }, { target: STEP * 3, duration: HOLD },
        { target: STEP * 4, duration: RAMP }, { target: STEP * 4, duration: HOLD },
        { target: STEP * 5, duration: RAMP }, { target: STEP * 5, duration: HOLD },
        { target: STEP * 6, duration: RAMP }, { target: STEP * 6, duration: HOLD },
      ],
    },
  },
  thresholds: {
    http_req_failed: ["rate<0.005"],
    handler_duration_ms: ["p(95)<250"],
    dropped_iterations: ["count<1"],   // generator must keep up or the run is void
  },
};

export function handleSummary(data) {
  const m = data.metrics;
  const line =
    `achieved ${m.http_reqs.values.rate.toFixed(1)} req/s | ` +
    `handler p95 ${m.handler_duration_ms.values["p(95)"].toFixed(1)} ms | ` +
    `queue ${m.receiver_queue_depth.values.value} | ` +
    `errors ${(m.http_req_failed.values.rate * 100).toFixed(2)}%\n`;
  return { "summary.json": JSON.stringify(data), stdout: line };
}

Run one level per invocation so each step produces its own summary file, which makes the analysis in Step 4 trivial and lets you abandon the run as soon as a step clearly fails:

for n in 1 2 3 4 5 6; do
  RATE=$((200 * n))
  STEP=$RATE HOLD=60s WEBHOOK_SECRET=$SECRET HOST=https://staging.example.com \
    k6 run --summary-export=/dev/null bench.js || true
  mv summary.json "step-${RATE}.json"
done

The dropped_iterations threshold is the guard that keeps the experiment honest. Any nonzero value means k6 wanted to issue requests and could not — the generator, not the receiver, was the constraint, and every number from that step is meaningless.

Step 4: Locate the knee and attribute it to a layer

The knee is where the latency curve stops being flat. Before it, added load is absorbed by idle capacity and latency barely moves; after it, each additional request per second waits behind the previous ones and latency rises superlinearly. Quote the last step before the knee as your capacity, not the highest rate that returned 2xx.

import json
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class StepResult:
    offered: float
    achieved: float
    handler_p95_ms: float
    error_rate: float
    queue_depth: float


def load_step(path: Path, offered: float) -> StepResult:
    metrics = json.loads(path.read_text())["metrics"]
    return StepResult(
        offered=offered,
        achieved=metrics["http_reqs"]["values"]["rate"],
        handler_p95_ms=metrics["handler_duration_ms"]["values"]["p(95)"],
        error_rate=metrics["http_req_failed"]["values"]["rate"],
        queue_depth=metrics["receiver_queue_depth"]["values"]["value"],
    )


def find_knee(
    steps: list[StepResult],
    latency_multiple: float = 2.0,
    admission_floor: float = 0.95,
    error_budget: float = 0.005,
) -> StepResult:
    """Last step that still behaved; the next one broke one of three rules."""
    baseline = steps[0].handler_p95_ms
    for previous, current in zip(steps, steps[1:]):
        if current.handler_p95_ms > baseline * latency_multiple:
            return previous
        if current.achieved < current.offered * admission_floor:
            return previous
        if current.error_rate > error_budget:
            return previous
    return steps[-1]


def attribute(step: StepResult, queue_max: int) -> str:
    """Which layer owns the ceiling at this step?"""
    if step.queue_depth > queue_max * 0.5:
        return "downstream: admission outran the worker drain rate"
    if step.handler_p95_ms > 100 and step.queue_depth < queue_max * 0.1:
        return "receiver: verify-and-enqueue path is the bottleneck"
    return "neither saturated: raise the offered rate and rerun"


if __name__ == "__main__":
    rates = [200, 400, 600, 800, 1000, 1200]
    steps = [load_step(Path(f"step-{r}.json"), float(r)) for r in rates]
    knee = find_knee(steps)
    print(f"sustainable admission rate: {knee.achieved:.0f} req/s")
    print(f"handler p95 at that rate:   {knee.handler_p95_ms:.1f} ms")
    print(f"limiting layer:             {attribute(steps[-1], queue_max=10_000)}")

Attribution is the half of the exercise that produces an action item. A flat queue depth with rising handler latency means the receiver itself is the constraint — signature verification, JSON parsing, TLS termination, or a synchronous call that should not be on the request path. A flat handler latency with a monotonically rising queue means the receiver is fine and the workers are the ceiling; the fix is worker concurrency or downstream capacity, and until then the correct behaviour under overload is to shed rather than accept, which is what applying backpressure to webhook consumers covers. Publishing the resulting number as a documented limit lets senders configure token bucket rate limiting against a real figure instead of guessing.

Verification and testing

A benchmark you cannot reproduce is an anecdote. Before quoting a number, prove the measurement itself is sound.

First, verify the receiver is actually verifying. Run a deliberately mis-signed request and confirm it is rejected — if it is accepted, every previous run measured a path with no cryptography in it:

curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST https://staging.example.com/webhooks/orders \
  -H 'Content-Type: application/json' \
  -H 'X-Webhook-Timestamp: 1721898000' \
  -H 'X-Webhook-Signature: v1=deadbeef' \
  -d '{"id":"evt_bogus","type":"order.created.v1"}'
# Expect: 401

Second, prove the generator was not the bottleneck at the knee. Re-run the knee step from two hosts at half the rate each; if the combined result is materially better than the single-host result, the single host was the limit and the real knee is higher:

# Host A and host B simultaneously, each at half the knee rate.
STEP=400 HOLD=60s k6 run bench.js   # on each of two generator hosts

Third, assert the run is repeatable. Three consecutive runs of the knee step should agree within about 10% on handler p95; wider variance means something uncontrolled is in the loop — a noisy neighbour, an autoscaler moving underneath you, or a database cache that was warm on one run and cold on the next.

def test_knee_is_reproducible():
    runs = [load_step(Path(f"knee-run-{i}.json"), offered=800.0) for i in (1, 2, 3)]
    p95s = [r.handler_p95_ms for r in runs]
    spread = (max(p95s) - min(p95s)) / min(p95s)
    assert spread < 0.10, f"handler p95 varies by {spread:.0%} across runs"
    assert all(r.error_rate < 0.005 for r in runs)

Finally, confirm the queue drained. A step that ends with a queue depth well above where it started did not run at a sustainable rate, no matter what its latency percentiles say — extend the hold and watch whether depth stabilises or climbs without bound.

Failure modes and gotchas

• Signing cost measured as receiver cost. crypto.hmac runs on the generator’s CPU once per iteration, and at several thousand iterations per second it can consume more CPU than issuing the requests. If generator CPU exceeds roughly 70%, pre-compute a rotating pool of signed bodies in the init context and refresh it once per second — still inside the timestamp tolerance window, but with the hashing amortised. Always cross-check dropped_iterations after making this change.
• Closed-model executors reporting a fake ceiling. With constant-vus, throughput is mathematically vus / response_time, so a receiver that slows down simply receives less load and never reveals its breaking point. The curve looks reassuringly flat right up to the moment production falls over. Only arrival-rate executors offer load independent of response time.
• Benchmarking through a proxy you forgot about. A staging ingress with connection limits, a WAF doing body inspection, or a service mesh sidecar will cap throughput well below the application’s real ceiling, and the resulting number is about the proxy. Run the same benchmark directly against a receiver pod to quantify the delta before attributing the ceiling to your code.
• A queue depth that never comes back down. If depth rises during a step and stays flat afterwards, the workers are at exactly their drain rate and the system has no headroom for the retries a real outage produces. Treat sustained depth above about 10% of the queue bound as being over capacity even when latency and error rate both look fine; the debugging path from there is the same one used for debugging failed webhook deliveries.

Frequently Asked Questions

Related

• Simulating Webhook Traffic Spikes — the burst-shaped counterpart to this steady-state benchmark.
• Consumer-Driven Contract Tests for Webhooks — keeping the synthetic payloads schema-valid.
• Token Bucket Rate Limiting for Webhook Senders — turning the measured ceiling into a published limit senders honour.
• Load Testing Webhook Endpoints — the parent overview on capacity testing.