Local Webhook Development with Tunnels: Exposing localhost Safely
Iterating on a webhook handler is part of webhook testing and local development, and it begins with a structural problem: providers dispatch HTTP POST requests to a public URL, but the handler you are editing runs on localhost behind NAT and a firewall, unreachable from the public internet. A tunnel solves this by allocating a public hostname at an edge service and forwarding every inbound request down a persistent outbound connection to a local port. The provider sees a normal HTTPS endpoint; you keep your debugger, hot reload, and logs on your own machine. The discipline that matters is making the tunneled environment behave exactly like production — same signature check, same timestamp window, same rejection codes — so that code which works locally also works when it ships.
Tunnel Patterns: Ephemeral vs Persistent Hostnames
Two implementation patterns dominate, and the choice has real consequences for how often you re-register with a provider.
Ephemeral tunnels. Running ngrok http 8000 allocates a random hostname for the session. This is the fastest path for a one-off experiment, but the URL changes on every restart, so you must re-register it with the provider each time — tedious when the provider’s dashboard rate-limits endpoint updates. Use ephemeral tunnels for throwaway debugging.
Persistent named tunnels. Both ngrok (reserved domains) and cloudflared (named tunnels bound to a DNS record) give a stable hostname that survives restarts. cloudflared tunnel run my-dev routes a subdomain you control to a local port indefinitely, so you register the URL with the provider once. This is the right default for any integration you will touch over more than a day, and it lets a team share a stable staging-like address.
A third consideration is body fidelity. A signature check hashes the exact bytes of the request body, so the tunnel must forward the raw stream untouched. Both ngrok and cloudflared do this by default, but a misconfigured intermediate proxy that re-encodes JSON or strips a header will break verification in ways that look like a cryptography bug. When a locally-running handler rejects a payload the provider considers valid, suspect the transport before the code.
Reproducing Signature Verification Locally
The point of local development is to exercise the real security path, not a bypassed one. Your handler must run the same HMAC signature verification against a development secret that production runs against the live secret, and it must enforce the same timestamp tolerance that underpins replay attack prevention. The development secret is a different value from production — never tunnel to a handler holding production secrets — but the verification logic is byte-identical.
# local_handler.py — identical verification path to production, dev secret only
import hmac, hashlib, time, os
from fastapi import FastAPI, Request, HTTPException
app = FastAPI()
SECRET = os.environ["WEBHOOK_DEV_SECRET"].encode() # NEVER the prod secret
TOLERANCE = int(os.environ.get("WEBHOOK_TOLERANCE_SEC", "300"))
@app.post("/webhook")
async def webhook(request: Request):
raw = await request.body() # raw bytes the tunnel forwarded
header = request.headers.get("x-signature", "")
try:
parts = dict(p.split("=", 1) for p in header.split(","))
ts, sig = int(parts["t"]), parts["v1"]
except (KeyError, ValueError):
raise HTTPException(status_code=401, detail="malformed signature header")
if abs(time.time() - ts) > TOLERANCE: # reject stale / replayed events
raise HTTPException(status_code=403, detail="timestamp outside tolerance")
expected = hmac.new(SECRET, f"{ts}.".encode() + raw, hashlib.sha256).hexdigest()
if not hmac.compare_digest(expected, sig):
raise HTTPException(status_code=403, detail="signature mismatch")
print(f"[ok] verified delivery, {len(raw)} bytes") # your debugger lives here
return {"status": "accepted"}
TOLERANCE is the setting developers are most tempted to widen, because a captured payload signed an hour ago will not replay through the tunnel until they do. Widening it locally is exactly the change that hides a replay bug until production. Keep the window at the production value and re-sign the capture with a fresh timestamp instead — the deliveries that fall outside the window are the ones the handler is supposed to refuse.
Tunnel Latency, Provider Timeouts, and the Breakpoint Problem
A tunnelled delivery does not take one internet hop, it takes two: the provider reaches the edge nearest to it, and the edge then pushes the request down the agent’s long-lived connection to your machine, which may be on a domestic connection on another continent. In practice that adds somewhere between 40 ms and 150 ms of round trip before your handler has executed a single line, and considerably more if you are on a VPN that hairpins traffic through a distant concentrator. Measure it once with a handler that returns 200 immediately and no work: that number is the floor under every local experiment, and it is the amount of the provider’s timeout budget you have already spent.
That budget is smaller than most people assume. Providers typically abandon a delivery somewhere between 5 and 10 seconds and then schedule a retry, which creates the single most confusing local-development failure there is. You set a breakpoint, the provider’s timeout expires while you are reading a variable, the provider records the delivery as failed and retries it with the same event ID, and when you finally continue execution you process the event — and then process the retry as well. The observable symptom is a duplicated row locally plus a red delivery in the provider’s dashboard for a request your handler completed successfully. Nothing is wrong with the code; the debugger consumed the deadline.
The structural answer is to keep the synchronous path short even in development. Verify, persist, acknowledge, and do the interesting work asynchronously — then set the breakpoint in the worker, where you can pause for as long as you like without a deadline. When you genuinely need to step through the request path, stop using live provider deliveries and replay a captured one instead, which is exactly what inspecting and replaying webhook deliveries exists for: a replayed request has no impatient sender behind it. Whatever you do, add an idempotency guard keyed on the event ID before you start debugging, or you will spend an afternoon chasing duplicates that only exist because you paused.
Two edge-level limits round out this picture and both produce errors your handler never sees. Tunnel edges buffer and size-cap request bodies, so a provider that batches events into a multi-megabyte payload can be rejected at the edge with a 413 while your logs stay empty. Free tiers also cap requests per minute — a legitimate burst of forty deliveries can come back as 429 from the edge, which looks exactly like your handler rate-limiting the provider. When a delivery fails and there is no corresponding line in your application log, the edge is the first place to look, not the last.
Locking Down a Publicly Reachable Development Endpoint
A tunnel makes a process on your laptop reachable by anyone who knows the hostname, and hostnames are easier to learn than people expect. Named tunnels bound to a real domain appear in Certificate Transparency logs within minutes of issuing a certificate, and random ephemeral hostnames on well-known tunnel domains are swept continuously by opportunistic scanners. Assume the URL is public knowledge the moment it exists, then decide what an unauthenticated stranger can reach through it.
The answer is usually more than the webhook route. Whatever else is bound to that port comes along: interactive API documentation, a framework debug console, verbose stack traces that echo environment variables, an admin UI, and behind them your development database with whatever real data got copied into it. The webhook handler itself is comparatively safe because signature verification fails closed, but it is only one route on a server you exposed wholesale. Scope the tunnel to the handler’s path where the agent supports it, disable auto-generated documentation and debug middleware in any configuration that gets tunnelled, and treat “would I be comfortable if this were indexed” as the acceptance test.
Human-facing tunnel authentication — an identity provider in front of the hostname, as offered by both major agents — is a good control for a shared dev UI and a poor one for webhooks, because the provider cannot complete an interactive login. If you enable it, scope the policy to everything except the webhook path and let the signature be the authentication on that route. Where the provider publishes egress IP ranges, an allowlist at the tunnel edge is a genuinely useful second factor, though treat it as defence in depth rather than a primary control: ranges change, and a signature check that fails closed does not. Finally, tie the tunnel’s lifetime to your working session. A tunnel left running overnight is an unattended public entry point into a machine that also holds your SSH keys, and there is no upside to leaving it up while you sleep.
Corporate Proxies, TLS Interception, and Restricted Egress
Tunnels are usually described as if the developer machine had an unfiltered path to the internet. On a corporate network it does not, and the resulting failures are among the hardest to diagnose because the agent’s own connection is what breaks, so nothing ever reaches your application logs. The general pattern is worth internalising: a tunnel agent needs a long-lived outbound connection, and every piece of enterprise network equipment is designed to inspect, terminate or recycle exactly that.
| Symptom | Underlying control | Practical fix |
|---|---|---|
| Agent exits immediately with a certificate error | TLS-intercepting proxy presents its own CA, which the agent does not trust | Point the agent at the corporate CA bundle, or have the tunnel hostnames added to the interception bypass list |
| Agent never connects, no error beyond a retry loop | Outbound UDP or a non-443 control port is blocked by the firewall | Force the agent onto HTTP/2 over 443 (cloudflared --protocol http2); ask for the documented endpoints to be allowed |
| Connection established, drops every few minutes | Deep packet inspection or a stateful firewall recycles long-lived sessions | Shorten nothing on your side — request an idle-timeout exemption, and expect ephemeral URLs to churn until you get it |
| Everything works until the VPN connects | Split-tunnel policy hairpins traffic through a distant concentrator, or blocks it outright | Exclude the tunnel agent from the VPN route, or accept the added round trip and raise your local timeouts |
| Large payloads fail while small ones succeed | MTU mismatch on a VPN or overlay network fragmenting the forwarded stream | Lower the interface MTU (1400 is a safe starting value) and retest with a payload above 8 KB |
The instinct to work around these controls locally — disabling certificate verification, installing an unmanaged proxy — is the wrong move, and not only for policy reasons: an intercepting proxy that re-encodes the request body will break signature verification in a way that looks precisely like a cryptography bug, so you would be trading a connectivity problem for a much more expensive one. The honest options are to get the agent’s documented endpoints allowed, or to stop needing a tunnel at all by running the handler somewhere the provider can already reach and driving it with signed fixtures instead.
Surviving URL Churn Across a Team
Hostname instability stops being an annoyance and becomes a coordination problem the moment more than one engineer works on the same integration. With five developers on ephemeral tunnels that restart perhaps four times a day, that is twenty endpoint edits a day in a provider dashboard that often rate-limits configuration changes, and every stale registration keeps receiving deliveries. That last detail is the one that bites: providers commonly disable an endpoint automatically after a run of consecutive failures, so one developer who closed their laptop with a registration still active can trip the account-wide auto-disable and silently kill the integration for everyone. The mechanism is the same one described in auto-disabling failing webhook endpoints, which is worth reading from the sender’s side to understand what your dead tunnel looks like to them.
Two arrangements solve this properly. The first is one wildcard DNS zone with a named tunnel per engineer — dev-ana.example.com, dev-luis.example.com — each registered once as its own endpoint and, where the provider supports it, scoped to a subset of event types or a per-developer test tenant so nobody receives everyone else’s traffic. The second is a shared fan-in receiver: a single stable endpoint in staging that verifies, stores and then re-posts each delivery to whichever developer tunnels are currently registered in a small dispatch table. It costs a service to run, but it gives the whole team replay for free, keeps exactly one registration in the provider’s dashboard, and means a developer going offline degrades to a failed re-post instead of a failed delivery.
If you stay on per-developer registrations, automate the update so nobody edits a dashboard. The ngrok agent exposes its current public URL on a local API, so a few lines at startup can read it and push it to the provider, making the URL change a non-event:
# sync_endpoint.py — run after the tunnel starts; re-registers the current URL
import os
import sys
import httpx
NGROK_API = "http://127.0.0.1:4040/api/tunnels"
PROVIDER_API = os.environ["PROVIDER_API"] # e.g. https://api.example.com
PROVIDER_TOKEN = os.environ["PROVIDER_TOKEN"] # test-mode key only
ENDPOINT_ID = os.environ["PROVIDER_ENDPOINT_ID"] # the registration to update
HANDLER_PATH = os.environ.get("HANDLER_PATH", "/webhook")
def current_public_url() -> str:
"""Read the https tunnel the local agent is currently serving."""
tunnels = httpx.get(NGROK_API, timeout=5.0).json()["tunnels"]
for tunnel in tunnels:
if tunnel["proto"] == "https":
return tunnel["public_url"]
raise RuntimeError("no https tunnel is running; start the agent first")
def main() -> int:
url = current_public_url() + HANDLER_PATH
resp = httpx.patch(
f"{PROVIDER_API}/v1/webhook_endpoints/{ENDPOINT_ID}",
headers={"Authorization": "Bearer " + PROVIDER_TOKEN},
json={"url": url},
timeout=10.0,
)
if resp.status_code >= 300:
print(f"[fail] provider rejected the update: {resp.status_code} {resp.text}")
return 1
print(f"[ok] endpoint {ENDPOINT_ID} now points at {url}")
return 0
if __name__ == "__main__":
sys.exit(main())
Wire that script into the same command that starts the tunnel and the app, and add a shutdown hook that either deletes the registration or points it at a known-dead placeholder. The deregistration half matters more than the registration half: it is what prevents your closed laptop from generating the consecutive failures that disable the endpoint for the whole team.
Environment Configuration and CI Hygiene
Local tunneling lives or dies on configuration discipline. Keep development secrets out of the repository and load them from a .env file that is git-ignored, with a committed .env.example documenting the required keys. Bind the tunnel to the same port your app listens on, and parameterize the tolerance window so tests can shrink it to assert that stale payloads are rejected.
# .env.example — committed; real .env is git-ignored
WEBHOOK_DEV_SECRET=replace-with-provider-test-secret
WEBHOOK_TOLERANCE_SEC=300
APP_PORT=8000
In CI you should not depend on a live tunnel — that introduces network flakiness and a third-party dependency into your pipeline. Instead, reserve tunnels for interactive local work and exercise the same handler in CI by posting locally-signed fixtures with a test client, which is exactly the deterministic approach used in webhook contract testing. The tunnel is for the human in the loop; CI runs the handler in-process.
The in-process substitute for a tunnel is covered in depth under webhook mocking and sandbox environments: rather than exposing a port to the internet, you stand up a mock webhook server for integration tests that signs deliveries with the same development secret and posts them at your handler over loopback. Keep the signing helper in one shared module so the tunnel path and the CI path cannot drift apart — a bug that only reproduces on one of the two is almost always two copies of the canonical signing string.
Failure Modes and Diagnostics
| Failure Mode | Root Cause | Mitigation |
|---|---|---|
| Signature mismatch only over the tunnel | An intermediate proxy re-encoded the JSON body before forwarding | Forward the raw byte stream; disable any body-rewriting in the tunnel config |
| Provider URL stops working after restart | Ephemeral tunnel allocated a new random hostname | Use a reserved domain (ngrok) or named tunnel (cloudflared) for a stable URL |
| Handler accepts unsigned payloads locally | Verification disabled “for convenience” during development | Always run the production verification path; use a dev secret, never skip the check |
| Replayed test event is accepted twice | Timestamp tolerance too wide or no idempotency check | Enforce the production tolerance window and an idempotency key on the handler |
502 from the tunnel edge |
Local app not listening on the forwarded port | Confirm the app port matches the tunnel target before registering the URL |
Debugging Checklist
- Tunnel forwards the raw, unmodified request body
- Handler verifies signatures with a development secret, not the production one
- Timestamp tolerance matches production and rejects stale events
- Public URL is stable (reserved domain or named tunnel) for multi-day work
- Secrets load from a git-ignored
.env, with a committed.env.example - An idempotency key prevents double-processing of replayed test events
For the exact commands to install a tunnel, expose your port, register the URL, and replay a provider’s test events end to end, follow testing webhooks locally with ngrok and tunnels.
Frequently Asked Questions
Can I point a provider's live endpoint at my laptop for one quick reproduction?
Treat that as a production change, because it is one. Real events carrying customer data land on an unmanaged disk, the live signing secret has to be present on that machine, and the moment you close the lid the provider starts recording consecutive failures that can auto-disable the endpoint for everyone. Capture the delivery in production, then replay the captured bytes at your local handler with a development secret instead.
Why does my own signed curl succeed while the provider's identical payload gets a 403?
Almost always because something between the wire and your hashing changed the bytes. A framework that parses JSON and re-serialises it will reorder keys and normalise whitespace; a middleware that transparently decompresses a gzipped body changes the length; and a handler that reads the body twice may hash an empty buffer the second time. Log the exact byte length and a hash of the raw body at the top of the handler and compare it against what the provider says it sent.
Does the local handler need TLS of its own?
No, and adding it usually causes problems. The edge terminates TLS on the public hostname and forwards the request to the agent inside its own encrypted connection, so the last hop to your port is plain HTTP over loopback. If you put a self-signed certificate on the local port, the agent will refuse to verify it unless you explicitly configure otherwise, and you will have added a failure mode that has no production counterpart.
How do I exercise the provider's retry behaviour through a tunnel?
Add a development-only route or flag that makes the handler return a chosen status code or sleep past the provider's timeout, then watch the intervals between the redeliveries that follow. The one caution is that deliberate failures count towards whatever consecutive-failure threshold the provider uses to disable an endpoint, so keep the experiment short and re-enable success before you walk away.
Can two engineers share one reserved domain or named tunnel?
Not simultaneously — the hostname is bound to whichever agent connects first, and the second agent will fail to start or take the route away from the first. Give each engineer their own subdomain under a wildcard record, which costs nothing extra and also lets each person filter to their own test tenant. Sharing one hostname by taking turns works for an afternoon and generates confusing cross-delivery bugs the rest of the time.
Deliveries started returning 403 after my laptop woke from sleep — why?
Suspended machines and containers frequently resume with a drifted system clock, and a handler whose clock is minutes behind the signer will reject perfectly valid signatures as out of tolerance. Print the delta between the header timestamp and the local clock in the rejection log so the cause is visible immediately, and force a time resync after resume rather than reaching for the tolerance setting. The same failure appears in virtual machines that were paused overnight.
Related
- Testing webhooks locally with ngrok and tunnels — the step-by-step command walkthrough.
- Webhook contract testing — deterministic, network-free payload assertions for CI.
- Inspecting and replaying webhook deliveries — capture and re-run failed deliveries.
- Webhook mocking and sandbox environments — what to run in CI when a tunnel is not available.
- Webhook Testing & Local Development — the full testing pipeline.