Testing Against Provider Sandbox Environments

A provider sandbox is the only test level where the payload is signed by the provider’s real signing code, delivered by the provider’s real dispatcher, to an endpoint the provider really resolved over DNS and TLS. That makes it the top rung of the ladder in webhook mocking and sandbox environments and the only place certain classes of defect can surface at all: an endpoint registered against the wrong event types, a subscription that was never activated, a signing secret copied from the wrong tenant, a middlebox stripping a header. None of that is reachable from recorded fixtures, because fixtures assume the delivery arrived.

It is also the level most likely to be trusted too much. Test modes are engineered for developer convenience, not for fidelity: they often do not retry, they run on quieter infrastructure, they emit placeholder values where production carries real data, and they will happily accept a single event at a leisurely pace when production will hand you four hundred in a burst. The discipline is to use the sandbox for exactly what only it can prove, then write down — explicitly, as a list with owners — everything it did not prove, and cover that in production.

Prerequisites

• A sandbox or test-mode account with its own API credentials, its own endpoint registration, and its own signing secret, none of which are shared with production.
• A publicly reachable staging endpoint, or a tunnel to a local handler when iterating on a laptop.
• The provider’s trigger mechanism: a CLI (stripe trigger), a redeliver button, or an API call that produces a real state change and therefore a real event.
• A delivery store the test can poll, so an assertion waits on a recorded delivery rather than on sleep.
• Zod 3 (or equivalent) for a configuration schema that fails at boot rather than at first request.
• Secrets loaded from a manager, not a checked-in .env — see storing webhook secrets in a secrets manager.

Step 1: Isolate sandbox configuration from production

Nearly every serious sandbox incident is a configuration crossover: a test key pointed at the production endpoint, a production key loaded by a staging deploy, or a sandbox webhook secret left in place after cutover so live deliveries fail verification. The defence is to treat environment configuration as one indivisible bundle. Credentials, endpoint URL, signing secret and database are chosen together by a single APP_ENV value, and any mismatched combination refuses to start.

// src/config.ts — one bundle per environment, validated at boot
import { z } from "zod";

const schema = z
  .object({
    APP_ENV: z.enum(["local", "sandbox", "production"]),
    PROVIDER_API_KEY: z.string().min(20),
    WEBHOOK_SECRET: z.string().min(20),
    WEBHOOK_ENDPOINT_URL: z.string().url(),
    DATABASE_URL: z.string().url(),
  })
  .superRefine((cfg, ctx) => {
    const isLiveKey = cfg.PROVIDER_API_KEY.startsWith("sk_live_");
    const isProdHost = new URL(cfg.WEBHOOK_ENDPOINT_URL).host.endsWith("api.example.com");

    if (cfg.APP_ENV === "production" && !isLiveKey) {
      ctx.addIssue({ code: "custom", message: "production env loaded a non-live provider key" });
    }
    if (cfg.APP_ENV !== "production" && isLiveKey) {
      ctx.addIssue({ code: "custom", message: "non-production env loaded a LIVE provider key" });
    }
    if (cfg.APP_ENV !== "production" && isProdHost) {
      ctx.addIssue({ code: "custom", message: "non-production env points at the production endpoint" });
    }
    if (cfg.APP_ENV === "sandbox" && cfg.DATABASE_URL.includes("prod")) {
      ctx.addIssue({ code: "custom", message: "sandbox env points at a production database" });
    }
  });

export const config = schema.parse(process.env);   // throws before the server binds a port

The rules read as paranoid until the first time one fires during a deploy. Add the equivalent assertion for the signing secret too: sandbox and production secrets must be different values, and a boot that finds them equal should fail rather than warn, since that state means one environment can forge deliveries the other will accept.

Step 2: Trigger test events and wait for the delivery

A sandbox test has two halves that people routinely conflate: causing an event and observing its delivery. Cause it through the provider’s own surface — a CLI trigger, a test-mode API call that creates a real sandbox object, or a redelivery of a past event — so the provider decides what payload to emit. Then observe by polling your delivery store for a matching record. Never assert with a fixed sleep: sandbox latency is variable, and a sleep long enough to be reliable makes the suite unbearable.

// test/sandbox/smoke.test.ts — runs only when sandbox credentials are present
import { describe, expect, it } from "vitest";
import { config } from "../../src/config";
import { pool } from "../../src/db";

async function triggerInvoicePaid(): Promise<string> {
  const res = await fetch("https://api.provider.test/v1/invoices", {
    method: "POST",
    headers: {
      authorization: `Bearer ${config.PROVIDER_API_KEY}`,
      "content-type": "application/json",
      "idempotency-key": `smoke-${Date.now()}`,
    },
    body: JSON.stringify({ customer: "cus_sandbox_1", amount_cents: 4200, pay_now: true }),
  });
  if (!res.ok) throw new Error(`trigger failed: ${res.status} ${await res.text()}`);
  return (await res.json()).id as string;
}

async function waitForDelivery(objectId: string, deadlineMs = 60_000) {
  const started = Date.now();
  while (Date.now() - started < deadlineMs) {
    const { rows } = await pool.query(
      `SELECT event_type, response_status FROM webhook_deliveries
       WHERE object_id = $1 ORDER BY received_at DESC LIMIT 1`,
      [objectId],
    );
    if (rows.length > 0) return rows[0];
    await new Promise((r) => setTimeout(r, 1_000));
  }
  throw new Error(`no delivery for ${objectId} within ${deadlineMs}ms`);
}

describe("provider sandbox", () => {
  it("delivers invoice.paid to the registered endpoint and we accept it", async () => {
    const invoiceId = await triggerInvoicePaid();
    const delivery = await waitForDelivery(invoiceId);
    expect(delivery.event_type).toBe("invoice.paid");
    expect(delivery.response_status).toBe(200);   // the provider's signature verified
  }, 90_000);
});

That single test covers a surprising amount of ground: the endpoint is registered, the event type is subscribed, DNS and TLS resolve from the provider’s network, the provider’s signature validates against your stored secret, and your handler returns a status the provider considers success. It is also slow, stateful and dependent on someone else’s uptime, which is exactly why it belongs on a schedule and a pre-release gate rather than in the pull-request path.

Step 3: Name every sandbox divergence explicitly

Sandbox behaviour diverges from production in ways that are individually reasonable and collectively dangerous, because each divergence quietly moves a risk from “tested” to “assumed”. Write them down as a table with an owner and a covering test; an unnamed divergence becomes an incident report later.

The compensations are already in your toolkit. Retry semantics belong to the mock sender, which can replay an attempt sequence in milliseconds and assert idempotency. Payload completeness belongs to captured fixtures, since only production traffic carries the unicode names, twelve-line addresses and null optional objects that break parsers. Volume and latency belong to load testing, driven at the throughput production actually reaches. One more divergence deserves its own mention: sandbox endpoints are frequently registered by hand and never re-verified, so include the endpoint registration and validation path in the sandbox run rather than assuming it still matches production.

Step 4: Close the remaining gaps in production

Some properties simply have no test-mode equivalent, and pretending otherwise is how launches go wrong. Prove them in production, deliberately and on a schedule: a canary event immediately after deploy, then a watched window with alerting on the metrics that would reveal a silent failure.

The canary is a small, reversible real transaction — a one-cent charge you refund, a subscription on an internal account, a repository event on a throwaway repo — that produces a genuine production event. It proves the live signing secret matches, the live endpoint is registered, and the live payload parses. The watched window is instrumentation, not attention: assert on delivery counts per event type against the previous week, on the retry ratio, and on the delivery latency percentiles that reveal a handler slowing under real volume.

// tools/post-deploy-check.ts — run 15 minutes after every production deploy
import { pool } from "../src/db";

interface Health {
  eventType: string;
  received: number;
  baseline: number;
  failureRatio: number;
}

export async function postDeployCheck(): Promise<Health[]> {
  const { rows } = await pool.query<Health>(
    `SELECT event_type                                            AS "eventType",
            count(*) FILTER (WHERE received_at > now() - interval '15 minutes')  AS received,
            count(*) FILTER (WHERE received_at BETWEEN now() - interval '7 days'
                                                   AND now() - interval '7 days' + interval '15 minutes') AS baseline,
            avg(CASE WHEN response_status >= 400 THEN 1.0 ELSE 0.0 END)
              FILTER (WHERE received_at > now() - interval '15 minutes')         AS "failureRatio"
       FROM webhook_deliveries
      GROUP BY event_type`,
  );

  const regressions = rows.filter(
    (r) => r.failureRatio > 0.01 || (r.baseline > 20 && r.received < r.baseline * 0.5),
  );
  if (regressions.length > 0) {
    console.error("post-deploy webhook regression", regressions);
    process.exitCode = 1;   // fail the pipeline step and hold the rollout
  }
  return regressions;
}

Pair that one-shot check with standing alerting on webhook delivery failures, because the interesting failures after a sandbox-green deploy are rarely loud: an event type that stops arriving looks exactly like a quiet afternoon until you compare it to last week.

Verification and testing

Guard the sandbox suite so it can never run with production credentials, and so a missing sandbox credential skips rather than silently passes:

// test/sandbox/guard.test.ts
import { describe, expect, it } from "vitest";

const hasSandbox = Boolean(process.env.PROVIDER_API_KEY?.startsWith("sk_test_"));

describe.runIf(hasSandbox)("sandbox guard", () => {
  it("never runs against a live provider key", () => {
    expect(process.env.PROVIDER_API_KEY?.startsWith("sk_live_")).toBe(false);
  });

  it("uses a webhook secret distinct from the production one", () => {
    expect(process.env.WEBHOOK_SECRET).not.toBe(process.env.PROD_WEBHOOK_SECRET_FINGERPRINT);
  });
});

Schedule the suite nightly and as a pre-promotion gate, and publish its result where the release process can see it. A sandbox run that has been red for a week and nobody noticed is worse than no sandbox run, because the release checklist still claims it is covered.

Failure modes and gotchas

• Sandbox timings used to size production. A test mode that acknowledges in 40 ms with no queue depth tells you nothing about a Monday-morning burst. Take capacity numbers from production traffic and load tests, and treat sandbox latency as a functional signal only.
• Assuming production retry behaviour was tested. Many sandboxes never retry, so a handler that returns 500 in test mode simply loses the event and the suite still passes. Prove retry and duplicate handling locally, where you control the attempt sequence, then confirm the provider’s real policy from its documentation and your production delivery log.
• Sanitised sandbox payloads hiding parser bugs. Test-mode objects carry tidy names, ASCII addresses and populated optional fields. The null middle name, the emoji in a company name, and the missing tax_id all arrive first in production. Cover them with captured and redacted fixtures rather than hoping the sandbox produces them.
• Shared or stale sandbox state. Sandboxes accumulate objects from every engineer who ever tested, and a suite that queries “the latest invoice” will eventually assert against someone else’s experiment. Create the object your test needs inside the test, key assertions on its returned id, and never on a global “most recent” lookup.
• A sandbox endpoint left registered after cutover. Forgotten sandbox registrations keep delivering to a staging service that may be running old code, holding stale secrets, or exposed publicly with no owner. Include registered endpoints in the same inventory review as production secrets, and delete them when the integration ships.

Frequently Asked Questions

Related

• Building a mock webhook server for integration tests — how to cover retries and duplicates the sandbox will not send.
• Recording and replaying webhook fixtures — capturing the real payload contents a sandbox sanitises away.
• Testing webhooks locally with ngrok and tunnels — reaching a laptop from the provider’s sandbox.
• Webhook mocking and sandbox environments — where sandbox testing sits among the other levels.