Configuring mTLS for Webhook Endpoints

This guide walks through enabling mutual TLS on a single webhook endpoint terminated by nginx and enforced again at the application, the concrete build behind Mutual TLS for Webhooks. The scenario: a payment provider dispatches webhooks to your /webhooks/payments endpoint, and you must reject any caller that cannot present a client certificate signed by your private CA — before the request reaches application code. Because mTLS authenticates the connection while payload signing authenticates the body, pair this with the message-level checks in Step-by-step HMAC webhook validation in Node.js for defense-in-depth.

Prerequisites

• A private certificate authority (root or intermediate) whose public certificate you control. This guide creates one in Step 1.
• nginx 1.25+ (or any TLS terminator that supports client-certificate verification) fronting your application.
• OpenSSL 3.x and curl built against OpenSSL for the verification steps.
• The webhook producer’s cooperation to install the issued client certificate, or your own out-of-band channel to deliver it.
• TLS 1.3 enabled end-to-end; disable TLS 1.0/1.1 entirely.

One prerequisite is architectural rather than a package version: confirm that nothing between the producer and nginx terminates TLS. A layer-7 cloud load balancer, a CDN in full-proxy mode, or a service mesh ingress will happily accept the connection, decrypt it, and open a fresh one to nginx without any client certificate — the endpoint then behaves exactly as it did before, which is why this misconfiguration usually ships unnoticed. Either put the balancer in TCP passthrough mode so the handshake reaches nginx intact, or move the client-verification configuration up to the balancer and have it forward the same identity headers Step 3 sets. Deciding this before you issue any certificates saves reissuing them against the wrong hostname later.

Step 1: Establish a private CA

Generate a CA key and self-signed CA certificate. Keep ca.key offline or in a hardware-backed store; it signs every client certificate you trust.

# CA private key (keep this secret and offline)
openssl genrsa -out ca.key 4096

# Self-signed CA certificate, valid 5 years
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 \
  -subj "/CN=Webhook Internal CA/O=Example" \
  -out ca.crt

ca.crt becomes the trust anchor that nginx uses to validate incoming client certificates. It must contain only CAs you intend to trust — never append public web-PKI roots.

Two choices in that command are worth understanding rather than copying. The 4096-bit RSA key is deliberately oversized for a root: it signs rarely, so the extra cost is irrelevant, and it will outlive several generations of leaf certificates. The five-year validity is a compromise — long enough that you are not re-bootstrapping trust every year, short enough that it will expire while someone who knows what it is still works there. Put a calendar reminder at four years, not four years and eleven months, because replacing a root means touching every verifier at once.

For anything beyond a single endpoint, do not sign leaves directly with this root. Create an intermediate CA, keep the root key offline — an encrypted volume in a safe, a hardware token, or an air-gapped host — and distribute only the intermediate to nginx. The practical benefit shows up during an incident: if the intermediate key leaks, you issue a new intermediate from the offline root and every verifier keeps working after a bundle swap, whereas a leaked root means rebuilding trust from scratch on every consumer. If your CA tooling supports name constraints, add them to the intermediate so it can only issue for identities under a namespace you control; that turns a stolen signing key from a fleet-wide compromise into a scoped one.

Step 2: Issue the client certificate

Create a key and certificate signing request (CSR) for the producer, then sign it with the CA. Encode the tenant identity in the subject so the application can map a verified connection to a tenant.

# Producer's key + CSR
openssl genrsa -out producer.key 2048
openssl req -new -key producer.key \
  -subj "/CN=payments-provider/O=Example/OU=tenant-42" \
  -out producer.csr

# Sign with the CA (90-day leaf; rotate before expiry)
openssl x509 -req -in producer.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -days 90 -sha256 -out producer.crt

# Confirm the chain validates
openssl verify -CAfile ca.crt producer.crt   # -> producer.crt: OK

Deliver producer.crt and producer.key to the webhook producer over a secure channel. Track the 90-day expiry now so rotation overlaps rather than failing closed — the cutover procedure is in rotating client certificates for mTLS webhooks. Every field of that subject has a job, and the OU is what Step 4 turns into a tenant.

Encoding tenancy in the certificate is what makes the rest of the design cheap. The alternative — matching on the common name in application code, or trusting a tenant identifier from the request body — either couples your authorization logic to a human-readable label that partners eventually ask you to change, or accepts an attacker-controlled value. The OU is a stable, CA-attested string that only the CA operator can set, so an authenticated caller cannot promote itself to another tenant by editing a payload. Where your tooling supports it, prefer a subject alternative name URI such as spiffe://webhooks/tenant/42 over the OU: SAN parsing is unambiguous, whereas distinguished-name string formatting varies between terminators and has produced more than one production mismatch where nginx emitted the components in a different order than the application expected.

The 90-day lifetime is the right default for a manually delivered certificate and the wrong one for an automated pipeline. Ninety days is short enough that a leaked key has a bounded useful life and long enough that a human rotation ritual runs four times a year rather than weekly. If you can automate issuance and reload — the consumer runs cert-manager, or you can hand them an ACME endpoint — drop to 24 or 72 hours instead, at which point revocation stops being a problem you need to solve. Whatever you pick, record the notAfter in the same inventory that holds your other expiring credentials on the day you issue it, because the certificate itself is the only other place that date exists and nothing queries it by default.

Step 3: Require client certificates at nginx

Configure the server block to terminate TLS, require a client certificate, validate it against ca.crt, and forward the verified identity to the upstream application.

server {
    listen 443 ssl;
    server_name hooks.example.com;

    ssl_certificate     /etc/nginx/certs/server.crt;
    ssl_certificate_key /etc/nginx/certs/server.key;
    ssl_protocols       TLSv1.3;

    # Require and verify the client certificate against our private CA.
    ssl_client_certificate /etc/nginx/certs/ca.crt;
    ssl_verify_client on;      # 'on' = mandatory; reject if absent/invalid
    ssl_verify_depth  2;

    location /webhooks/payments {
        # Reject anything nginx did not successfully verify.
        if ($ssl_client_verify != SUCCESS) { return 403; }

        # Forward the verified identity to the app for tenant mapping.
        proxy_set_header X-Client-Verify  $ssl_client_verify;
        proxy_set_header X-Client-Subject $ssl_client_s_dn;
        proxy_pass http://127.0.0.1:8080;
    }
}

With ssl_verify_client on, a caller presenting no certificate or an untrusted one fails during the TLS handshake — nginx never invokes the location block for it.

The if ($ssl_client_verify != SUCCESS) guard looks redundant next to on, and on this server block it is. Keep it anyway: it is what makes the block safe to copy into a server that later adds optional for a different path, and it is the single line that turns an accidental downgrade into a 403 instead of an accepted anonymous request. ssl_verify_depth 2 allows one intermediate between the leaf and the trusted root; raise it only if you actually sign through a longer chain, because a permissive depth widens what a compromised sub-CA can present.

Choosing the right client verification mode

The ssl_verify_client directive has four values and they differ in where the rejection happens, which changes what you can observe and what you can roll back. on rejects at the TLS layer, so there is no request to log and no HTTP status to chart — excellent for steady state, useless for finding out who would break before you enforce. optional still fails the handshake for an untrusted certificate but lets a caller with no certificate through, setting $ssl_client_verify to NONE; that is the mode to run during a rollout, paired with logging rather than rejection. optional_no_ca accepts anything and defers the entire trust decision to your application, which is only appropriate when a downstream service holds the trust store.

Sequence the change accordingly: deploy with optional and a log line recording $ssl_client_verify per request, watch for a full week including a low-traffic weekend, and only flip to on once every expected identity shows SUCCESS on every call. Keep the directive in a small include file so the rollback is a one-line edit plus nginx -s reload, which takes about a second and drops no in-flight requests. Baking the mode into a container image turns that rollback into a redeploy, and a redeploy during a total delivery outage is the wrong time to discover your image build takes twelve minutes.

Step 4: Enforce identity in the application

Never trust the network alone: re-check the forwarded headers and resolve the subject to a tenant before processing. This also guards against a misconfiguration where the headers arrive from somewhere other than your trusted proxy.

from flask import Flask, request, abort

app = Flask(__name__)

# Map verified certificate subjects to tenant IDs.
SUBJECT_TO_TENANT = {
    "CN=payments-provider,O=Example,OU=tenant-42": "tenant-42",
}

@app.post("/webhooks/payments")
def payments_webhook():
    if request.headers.get("X-Client-Verify") != "SUCCESS":
        abort(403, "client certificate not verified")

    subject = request.headers.get("X-Client-Subject", "")
    tenant = SUBJECT_TO_TENANT.get(subject)
    if tenant is None:
        abort(403, "unknown client certificate subject")

    # Connection identity established. Now verify the payload signature
    # (HMAC/JWT) before mutating state.
    request.environ["tenant_id"] = tenant
    return ("", 204)

Three properties of that handler are load-bearing. It rejects rather than defaults when the subject is unknown, so adding a partner is an explicit map change instead of a silent grant. It checks the verify header before reading the subject, so a proxy that forwards a subject without a verify result cannot slip through. And it stops at establishing identity — the payload signature check still has to run before any state change, because a correctly authenticated connection can still carry a body that was rewritten inside your own network or replayed from an earlier capture.

The map itself is the piece that outgrows the code fastest. A dictionary is right for a handful of partners: changes go through review, and the deploy is your audit log. Once onboarding happens more often than quarterly, move it to a table keyed on the exact distinguished-name string with the tenant as the value, and cache it in process with a short refresh interval so a database outage does not become a delivery outage. Whichever storage you choose, normalise the DN the same way on both sides — nginx renders $ssl_client_s_dn in RFC 2253 order, which reverses the component order some other terminators use, and a mismatch here presents as every request being rejected with a subject that looks correct to the human reading the log.

Verification and testing

Confirm an authenticated call succeeds and an unauthenticated one is rejected.

# 1. Inspect the handshake; expect "Verify return code: 0 (ok)".
openssl s_client -connect hooks.example.com:443 \
  -cert producer.crt -key producer.key -CAfile ca.crt -tls1_3 </dev/null

# 2. Authenticated request -> 204
curl -sw '%{http_code}\n' https://hooks.example.com/webhooks/payments \
  --cert producer.crt --key producer.key --cacert ca.crt \
  -d '{"event":"payment.succeeded"}'

# 3. No client certificate -> handshake aborts / 400-level, NOT 204
curl -sw '%{http_code}\n' https://hooks.example.com/webhooks/payments \
  --cacert ca.crt -d '{}' || echo "rejected as expected"

In CI, assert the third command fails: a test that authenticated traffic passes is incomplete without a test that anonymous traffic is refused.

Add a fourth case that most teams skip — a certificate signed by a different CA. Generate a throwaway root and leaf once, commit them as test fixtures, and assert that the call is rejected. Without it, a regression that swaps ssl_client_certificate for a bundle including a public root passes every other test in the suite, because your legitimate certificate still validates. Round the suite out with an expiry assertion so a certificate that is about to lapse fails the build rather than production:

# 4. A leaf from an untrusted CA must be refused, not merely logged.
curl -sw '%{http_code}\n' https://hooks.example.com/webhooks/payments \
  --cert rogue.crt --key rogue.key --cacert ca.crt -d '{}' \
  && echo "FAIL: untrusted leaf accepted" || echo "rejected as expected"

# 5. Fail the build if the client leaf expires within 21 days (1814400 seconds).
openssl x509 -in producer.crt -noout -checkend 1814400 \
  || { echo "FAIL: producer.crt expires within 21 days"; exit 1; }

# 6. Print the exact subject nginx will forward, to seed SUBJECT_TO_TENANT.
openssl x509 -in producer.crt -noout -subject -nameopt RFC2253

Run cases 5 and 6 on a schedule as well as on every commit. A build that only runs when someone changes code will not warn you about a certificate expiring during a quiet quarter, which is precisely when the expiry will land.

Failure modes and gotchas

Almost every mTLS defect resolves to one binary question — did the handshake complete? — and the branch you land on tells you whether to look at the certificate, the trust chain, or the proxy headers.

• 400 No required SSL certificate was sent — the client did not present a certificate. Confirm the producer is actually loading producer.key/producer.crt, and that no intermediate proxy is stripping the TLS session.
• SSL certificate verify failed on a valid certificate — nginx’s ssl_client_certificate bundle is missing the issuing CA or an intermediate. Append the full chain to ca.crt and raise ssl_verify_depth if you sign through an intermediate.
• Headers spoofable when the app is reachable directly — if a client can hit the app on port 8080 bypassing nginx, it can forge X-Client-Verify. Bind the app to localhost only, and strip these headers at the proxy edge for any path nginx did not verify.
• Silent expiry outage — a lapsed leaf certificate rejects 100% of deliveries at the handshake with no application log. Alert on notAfter approaching, and rotate with overlap as described in Mutual TLS for Webhooks.
• Keep-alive outliving revocation — nginx evaluates the client certificate at handshake time, not per request. A connection established before you revoked a certificate keeps delivering until it closes, so a revocation you believe is instant can lag by your full keepalive_timeout. Set that timeout deliberately (60 seconds is a reasonable ceiling for a webhook endpoint) and, for an urgent revocation, restart the workers rather than reloading.
• Distinguished-name formatting mismatch — the subject string nginx forwards is RFC 2253 formatted and comma-separated with no spaces, which is not the order or spacing openssl req -subj accepts as input. Copying the -subj argument straight into your tenant map produces a permanent 403 that looks like a typo nobody can find. Derive the map entry from openssl x509 -noout -subject -nameopt RFC2253 instead.
• Certificate lacks the client-authentication key usage — a leaf issued with openssl x509 -req and no extension file carries no extended key usage at all, which most terminators tolerate but some strict clients and mesh proxies reject outright. If the handshake fails only for one consumer’s stack, inspect the leaf with openssl x509 -noout -text and add extendedKeyUsage = clientAuth through an extensions file when signing.

Frequently Asked Questions

Related

• Rotating client certificates for mTLS webhooks — replacing the 90-day leaf issued in Step 2 without a delivery gap.
• Step-by-step HMAC webhook validation in Node.js — payload-level signing to pair with connection-level mTLS.
• Storing webhook secrets in a secrets manager — where the CA key and leaf private keys belong once you leave the laptop.
• Mutual TLS for Webhooks — the trust model, rotation, and pinning concepts behind this configuration.