Webhook Contract Testing: Pinning the Payload Shape in CI

Catching a breaking payload change before it ships is the central concern of webhook testing and local development, and contract testing is the cheapest, most deterministic tool for the job. A webhook payload is a contract between a producer you frequently do not control and a consumer you do; when the producer adds a required field, renames an enum, or changes a date format, your handler can keep returning 200 OK while silently writing corrupt state. Contract testing makes that implicit agreement explicit and executable: you encode the exact shape your consumer depends on, then assert against it on every change so a drift fails a build rather than an invoice. Unlike end-to-end tests, contract tests need no network, no tunnel, and no provider sandbox — they run in milliseconds, which is what lets you gate every commit on them.

The Contract Surface: What the Consumer Actually Depends On

A payload contract is not the provider’s payload. It is the projection of that payload onto the fields your handler reads, and getting that projection right is the difference between a suite that catches real breaks and one that cries wolf every time the provider ships a feature. Four properties are worth distinguishing, because each one fails differently and only three of them are reachable by a schema at all. Presence is whether the key is there; its failure is a TypeError on the first property access, usually inside a queue worker where nobody sees the stack trace. Type is whether amount_cents is a JSON number or the string "4200"; its failure is a silent coercion that stores NaN or concatenates instead of adding. Domain is enum membership, string format, and numeric range; its failure is a value your switch has no branch for, which in most codebases means the event is quietly dropped by the default case. Semantics is what the value means — and no schema keyword can see it.

That fourth axis is the one worth internalising before you write a line of schema. If a provider changes amount_cents from gross to net, the field is still present, still an integer, still non-negative, and still matches every pattern you wrote. Every contract test stays green while your ledger drifts by the tax amount on every invoice. Contract testing buys you presence, type, and domain cheaply and completely; semantics is covered by reconciliation checks and business invariants — a nightly job that asserts the sum of your recorded amounts matches the provider’s own reported totals — and it is worth writing that down explicitly so nobody mistakes a green contract suite for proof that the numbers are right.

The second decision is how much of the payload to pin. A mature provider’s invoice.paid body carries sixty to ninety fields; a typical handler reads six. Deriving the schema from the whole captured payload feels thorough and is actively harmful: every one of those eighty-odd unread fields becomes a tripwire that fires on changes you do not care about, and after the third false failure someone adds --update-snapshots to CI and the suite stops meaning anything. Derive the contract from the handler’s field reads instead. In practice that means grepping for every event.data. access, or better, funnelling parsing through a single typed decoder so the decoder’s type is the enumeration of what you depend on. The rule of thumb that survives contact with real codebases: if removing a field would not change any observable behaviour of your service, it does not belong in the contract.

Pattern 1: JSON Schema Assertions

The lightest-weight pattern asserts each incoming payload against a JSON Schema that captures the fields, types, and required keys your handler actually reads. This works even when the producer is a third party with no contract-testing tooling: you derive the schema from a real captured payload, commit it, and validate every fixture against it. The schema is the contract, and it pairs directly with the producer-side event schema design and payload versioning discipline.

// contract.schema.ts — the shape the consumer depends on
export const orderCreatedSchema = {
  $schema: "https://json-schema.org/draft/2020-12/schema",
  type: "object",
  required: ["type", "id", "data"],
  additionalProperties: false,   // a new top-level field fails the test loudly
  properties: {
    type: { const: "order.created" },
    id: { type: "string", pattern: "^evt_" },
    data: {
      type: "object",
      required: ["order_id", "amount_cents", "currency"],
      properties: {
        order_id: { type: "string" },
        amount_cents: { type: "integer", minimum: 0 },
        currency: { type: "string", pattern: "^[A-Z]{3}$" },
      },
    },
  },
} as const;

// contract.test.ts — Vitest, runs with no network
import Ajv from "ajv";
import { describe, it, expect } from "vitest";
import { orderCreatedSchema } from "./contract.schema";
import capturedPayload from "./fixtures/order_created.json";

const validate = new Ajv({ allErrors: true }).compile(orderCreatedSchema);

describe("order.created contract", () => {
  it("accepts the captured production payload", () => {
    const ok = validate(capturedPayload);
    expect(ok, JSON.stringify(validate.errors)).toBe(true);
  });

  it("rejects a payload missing a field the handler reads", () => {
    const { currency, ...incomplete } = capturedPayload.data;
    const ok = validate({ ...capturedPayload, data: incomplete });
    expect(ok).toBe(false);   // drift is caught here, not in production
  });
});

Setting additionalProperties: false is deliberate: it turns a silent additive change into a loud test failure, forcing a human to decide whether the new field matters. For deliberately permissive consumers, relax it per-object rather than globally.

Read the schema as four independent tripwires rather than one document. Each keyword catches a distinct class of upstream change, and dropping any one of them opens a specific blind spot — a schema with required but no pattern will happily accept an ID format migration that breaks your foreign keys.

Pattern 2: Consumer-Driven Pact Contracts

When you control both sides — or the producer is a cooperating internal team — a Pact-style consumer-driven contract is stronger. The consumer writes expectations as code, generating a contract file; the producer’s CI replays that contract against its real serializer and fails if it can no longer satisfy it. This catches drift on the producer’s build, before the change ever reaches a consumer.

// consumer.pact.test.ts — the consumer declares what it needs
import { MessageConsumerPact, synchronousBodyHandler } from "@pact-foundation/pact";
import { like, term } from "@pact-foundation/pact/src/dsl/matchers";

const messagePact = new MessageConsumerPact({
  consumer: "billing-service",
  provider: "orders-service",
});

describe("orders-service webhook contract", () => {
  it("emits an order.created event this consumer can process", () => {
    return messagePact
      .expectsToReceive("an order.created webhook")
      .withContent({
        type: "order.created",
        id: term({ generate: "evt_123", matcher: "^evt_" }),
        data: {
          order_id: like("ord_987"),
          amount_cents: like(4200),
          currency: term({ generate: "USD", matcher: "^[A-Z]{3}$" }),
        },
      })
      .verify(synchronousBodyHandler((event) => {
        // the real handler logic the consumer ships
        if (typeof event.data.amount_cents !== "number") throw new Error("bad amount");
      }));
  });
});

The matchers (like, term) assert types and patterns, not exact values, so the contract survives realistic payload variation while still failing on a renamed field or a changed format. The generated contract is published to a broker that the producer verifies against — the flow in the diagram above.

Which of the two patterns you reach for is decided almost entirely by who owns the producer, because that determines whose build can be made to fail. A schema over captured fixtures is the only option against a third party; a message pact is strictly stronger when the producer’s CI can be made to run your expectations.

Validation, Signatures, and the Contract Boundary

A contract test verifies shape, not authenticity — the two are complementary and must both run. Contract tests operate on already-verified payloads; they assume the HMAC signature verification gate has run first. Keep these layers ordered in the handler: verify the signature on the raw body, then parse, then assert the contract. Validating shape before authenticity would waste cycles on forged input and risk parsing hostile payloads. Use the captured fixtures from your inspecting and replaying webhook deliveries store as contract test inputs so the schema is grounded in payloads the provider really sent.

CI Gating

Contract tests earn their value only when they block merges. Run them as a required check on every pull request, and run the consumer suite again on a schedule against the latest captured fixtures so a provider’s quiet change surfaces within a day.

# .github/workflows/contract.yml
name: contract-tests
on: [pull_request]
jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: "20" }
      - run: npm ci
      - run: npm run test:contract   # vitest contract suite — required check

Because the suite needs neither a network nor a provider, it slots in next to the rest of your offline coverage under webhook mocking and sandbox environments: a mock webhook server for integration tests proves the handler survives real HTTP and duplicate deliveries, while the contract suite proves the body it received still has the shape the handler reads. Run them in the same job — a passing contract with a handler that cannot parse the request is not a green build.

Evolving a Contract Without a Flag Day

A contract that can only ever be tightened becomes a liability the first time the producer needs to ship a legitimate change. The mechanism that keeps both sides moving is expand-and-contract, and the only genuinely hard part is choosing how long the expanded state lasts. In the expand phase the new schema is added alongside the old one and both shapes validate; the producer starts emitting the new shape while continuing to populate the old field; consumers switch their reads over; and only in the contract phase does the old shape become invalid. Each phase is a separate deploy on each side, which is precisely the point — no single release has to land simultaneously in two repositories.

The window between expand and contract is where teams guess badly. It is not a matter of taste; it is arithmetic over how long an old-shaped payload can still be in flight. A delivery first attempted just before the cutover may still be retried for the length of the sender’s retry horizon — commonly 24 to 72 hours for a mature provider using exponential backoff. If your operations team can re-inject a delivery from the dead-letter queue, the oldest payload the handler can ever see is as old as your dead-letter retention, commonly 14 days. Add one release cycle so a consumer that missed the migration has a chance to catch up, and the arithmetic for a typical stack is 72 hours plus 14 days plus 7 days, or roughly 24 days — which is why 30 days is the default worth writing into the deprecation note rather than the 3 days someone will propose in review.

Running two schemas at once needs a dispatch layer, and the shape of that layer matters more than it looks. Keying only on the event type forces you to encode version differences inside one increasingly oneOf-heavy schema, which produces validation errors so vague they are useless in an incident. Keying on the pair of event type and payload version keeps each schema small and each error message specific, and it gives you the one behaviour every migration needs: an explicit, separately-metered outcome for a version you have never seen, rather than a generic validation failure that looks identical to a corrupt payload.

// contracts/registry.ts — one validator per (event type, payload version)
import Ajv, { ValidateFunction } from "ajv";
import { orderCreatedV1 } from "./order_created.v1.schema";
import { orderCreatedV2 } from "./order_created.v2.schema";

const ajv = new Ajv({ allErrors: true, strict: false });

const registry = new Map<string, ValidateFunction>([
  ["order.created:1", ajv.compile(orderCreatedV1)],
  ["order.created:2", ajv.compile(orderCreatedV2)],
]);

export type ContractResult =
  | { ok: true; version: number }
  | { ok: false; reason: "unknown_version" | "schema_violation"; detail: string };

export function checkContract(event: { type: string; version?: number }): ContractResult {
  // Payloads that predate versioning are v1 by definition, never "unknown".
  const version = event.version ?? 1;
  const validate = registry.get(`${event.type}:${version}`);
  if (!validate) {
    return { ok: false, reason: "unknown_version", detail: `${event.type} v${version}` };
  }
  if (!validate(event)) {
    return { ok: false, reason: "schema_violation", detail: ajv.errorsText(validate.errors) };
  }
  return { ok: true, version };
}

The unknown_version branch is the early-warning system: it fires the moment a producer starts emitting a version you have not modelled, before a single field-level assertion has a chance to be wrong. Alert on it separately from schema_violation, because the two demand different responses — an unknown version means someone shipped ahead of you, while a schema violation means someone shipped something different from what they said.

Fixture Drift and the Half-Life of a Captured Payload

A contract test is only as truthful as the payload it runs against, and fixtures rot. The provider ships a feature and a new enum value appears; a field that was always populated starts arriving null for customers in a new region; an internal identifier changes format for accounts created after a migration. None of this touches your repository, so the suite stays green while production starts failing — the single most common way a contract-tested integration still breaks, and the hardest to diagnose because CI is telling you everything is fine.

The fix is to close the loop with production traffic. Run the same compiled validator against live deliveries and emit the result as a metric rather than acting on it: a counter labelled by event type, schema version, and the failing keyword. This is shadow validation, and its cost is small enough to make sampling debates unnecessary — a compiled Ajv validator over a 4 KB payload runs in roughly 10 to 40 microseconds, so at 500 deliveries per second you are spending well under 2% of a single core to validate everything. Sample only if you are an order of magnitude above that, and if you do, sample by event type rather than uniformly, or the rare event types you understand least will be the ones you never observe.

Thresholds matter because provider rollouts are gradual. A change deployed to 1% of the provider’s fleet shows up as a 1% violation rate that looks exactly like noise on a dashboard scaled for outages. Alert on any sustained violation rate above 0.1% over 15 minutes for a given event type, which is low enough to catch a canary rollout on the day it starts and high enough to survive the occasional genuinely malformed payload. Pair it with a freshness rule in the suite itself: stamp every fixture with the timestamp it was captured, and fail — not warn — when the newest fixture for an event type is more than 90 days old. It is a deliberately annoying test, and the annoyance is the mechanism: it forces someone to look at a real recent delivery four times a year.

Contracts You Do Not Own

Everything above assumes you can eventually make somebody change something. Against a third-party provider you cannot fail their build, cannot review their diff, and frequently cannot get a straight answer about when a change ships. What you can control is detection latency — the interval between the first payload that violates your contract and the moment a human knows. Every tactic here is about shrinking that number.

Start by pinning the provider’s API version wherever one is offered, whether that is an account-level setting or a per-request version header. Pinning converts an ambient, unannounced change into a deliberate upgrade you schedule. It is not complete protection: providers routinely ship additive and bug-fix changes outside the versioning scheme, and a pinned version can be retired from under you. But it removes the largest category of surprise, and it makes the remaining surprises smaller. Next, record the provider’s version identifier as a dimension on every delivery you store. Without it, a change rolling out to 5% of the provider’s fleet presents as intermittent failures that correlate with nothing, and the usual misdiagnosis is a flaky network or one bad host on your own side; with it, the violation metric splits cleanly by version and the shape of the problem is obvious in one query.

When you do escalate, evidence decides how fast you get a response. A ticket saying “your payloads changed” gets triaged into a queue; a ticket with three delivery identifiers, exact timestamps, the JSON pointer of the offending field, and the before-and-after values usually reaches an engineer the same day. Keep the shadow validator’s error detail structured enough to paste. Finally, accept that the provider’s documentation is not the contract — the traffic is. When the docs promise a string and the wire delivers null, model the wire, add a comment naming the support ticket, and revisit it when they reply. A schema that encodes the documentation instead of reality is a schema that fails on every real delivery while remaining perfectly defensible in review.

When a Contract Breaks in Production

The alert fires at 09:40 and the shadow validator says 100% of charge.refunded deliveries are violating the schema. Work the problem in this order, because the first question determines everything after it.

1. Read the failing keyword before anything else. An additionalProperties failure means the provider added something and nothing of yours is broken — it is a Monday-morning schema widening, not an incident. A required, type, or enum failure means a field your handler genuinely reads has changed, and the clock is now running against the sender’s retry horizon.
2. Decide the disposition of the affected traffic. Rejecting is safe only if you will ship a fix inside the retry window, because a rejected delivery that exhausts its attempts is gone. Accepting and routing the payload to a dead-letter queue for later replay is the better default: it preserves the event, keeps the provider’s delivery success rate healthy, and turns a deadline into a backlog. Accepting and degrading — processing the fields you can still read and flagging the record — is appropriate only when partial processing is genuinely safe, which for anything touching money it is not.
3. Narrow the relaxation, never disable the check. The temptation is a feature flag that turns off validation. Instead, relax the single keyword for the single event type, in code, with an expiry date in the comment and a linked issue. A global bypass gets committed in ten minutes and removed in ten months.
4. Ship the schema change, the handler change, and the new fixture in one pull request. The fixture is the durable part: without a captured copy of the payload that broke you, the same regression returns the next time someone refactors the parser. This is also the moment to add whichever assertion would have caught it earlier — usually an enum you had left as a bare string.
5. Measure detection latency in the retrospective. First bad delivery to alert, in minutes. If it was hours, the shadow validator is sampling too thinly or the threshold is too high. If it was days, you learned about it from a customer, and the shadow validator is not deployed at all.

Worked through with real numbers, the common case looks like this: a provider adds a field on 100% of one event type, a closed schema rejects, the handler returns 400, and the provider retries with backoff eight times over 72 hours. At 5,000 events per day for that type, roughly 12,000 deliveries accumulate as pending retries before anyone finishes lunch. Widening additionalProperties for that one event type and deploying takes ten minutes, and the retry backlog drains on its own — which is the whole argument for a closed schema being loud rather than a permissive one being quiet.

Failure Modes & Mitigations

Debugging Checklist

• additionalProperties: false (or per-object equivalent) catches added fields
• Fixtures are captured real payloads, not hand-authored
• Matchers assert types and patterns, not brittle literal values
• Signature verification runs before shape assertion in the handler
• The contract suite is a required PR check and runs on a schedule
• Each schema is versioned alongside the payload version it describes

For the full producer/consumer walkthrough with a published broker and provider verification, see consumer-driven contract tests for webhooks.

Frequently Asked Questions

Related

• Consumer-driven contract tests for webhooks — the end-to-end Pact walkthrough.
• Local webhook development with tunnels — exercise contracts against live deliveries.
• Webhook mocking and sandbox environments — the offline coverage that runs beside the contract suite.
• Event Schema Design — the producer side of the payload contract.
• Webhook Testing & Local Development — the full testing pipeline.