Rotating client certificates for mTLS webhooks without dropped deliveries

Your monitoring finally noticed: the client certificate the delivery fleet presents to a partner’s webhook endpoint expires in 30 days. Unlike an expiring signing secret, a lapsed client certificate fails at the TLS handshake — no request body is ever sent, no application log line appears on either side, and every delivery in flight turns into a connection error at once. This guide is the rotation procedure that avoids that cliff, extending Mutual TLS for Webhooks and picking up exactly where Configuring mTLS for webhook endpoints stopped, at the 90-day leaf it told you to track. The shape of the fix mirrors zero-downtime webhook secret rotation: never have exactly one valid credential during a change. The mechanics differ, because a certificate has two independent things to overlap — the leaf’s validity period and the issuing CA’s presence in the verifier’s trust bundle — and getting their order backwards is the classic way to turn a routine rotation into an outage.

The whole procedure rests on one invariant: a certificate must be trusted everywhere before it is presented anywhere, and it must stay trusted until nothing presents it. Trust is what the receiver configures; presentation is what the sender does. If presentation moves first, every handshake using the new leaf fails with unknown ca until the trust rollout catches up.

Prerequisites

• A working mTLS endpoint, terminated by nginx 1.25+ or an equivalent proxy configured as in Configuring mTLS for webhook endpoints.
• OpenSSL 3.x on the machine that issues certificates, with access to the CA key (ideally in an HSM or a KMS-backed signer).
• Node.js 20+ with TypeScript 5.4 and undici 6 on the sending side — node:crypto’s X509Certificate and undici’s per-agent connect options do all the work here.
• A secret store that versions binary secrets (Vault, AWS Secrets Manager, or a Kubernetes Secret synced by external-secrets), so the key never travels through a build artefact. See storing webhook secrets in a secrets manager for the storage side.
• prom-client 15+ and an alerting pipeline, because unmonitored expiry is the failure this whole procedure exists to prevent.
• At least 21 days of remaining validity before you start. A rotation compressed into 48 hours skips the soak and is where mistakes hide.

Step 1: Issue the replacement with an overlapping validity window

Start by reading the facts off the certificate that is actually deployed, rather than the one in the repository — drift between them is common and it determines your real deadline.

# What is deployed right now: subject, issuer, and both validity bounds.
openssl x509 -in /etc/webhook/tls/client.pem -noout -subject -issuer -dates -serial

# Confirm the endpoint's expectation matches, straight off the wire.
openssl s_client -connect hooks.partner.example:443 \
  -cert /etc/webhook/tls/client.pem -key /etc/webhook/tls/client.key \
  -CAfile /etc/webhook/tls/server-ca.pem </dev/null 2>/dev/null \
  | openssl x509 -noout -dates

Generate a new key — rotation that reuses the old private key rotates nothing — and sign a leaf that starts today and outlives the old one. Keep the subject identical if the receiver maps subjects to tenants, because changing CN and rotating the key at the same time means two variables in one change.

# 1. New private key (never copy the old one).
openssl genrsa -out client-b.key 2048

# 2. CSR with the SAME subject the receiver already maps to a tenant.
openssl req -new -key client-b.key \
  -subj "/CN=payments-provider/O=Example/OU=tenant-42" \
  -out client-b.csr

# 3. Sign it. notBefore = now, notAfter = now + 90d, overlapping the old leaf by 30d.
openssl x509 -req -in client-b.csr -CA ca-b.crt -CAkey ca-b.key \
  -CAcreateserial -days 90 -sha256 \
  -extfile <(printf 'keyUsage=critical,digitalSignature,keyEncipherment\nextendedKeyUsage=clientAuth\n') \
  -out client-b.crt

# 4. Ship the leaf plus its issuing chain, leaf FIRST, so the server can build a path.
cat client-b.crt ca-b.crt > client-b-fullchain.pem
openssl verify -CAfile ca-b.crt client-b.crt   # -> client-b.crt: OK

If you are rotating only the leaf and keeping the same CA, ca-b.crt is ca.crt and Step 2 is a no-op verification. If the CA itself is being replaced — because it is expiring, or moving into a KMS — the two overlaps must be nested: the CA overlap opens first and closes last.

Step 2: Trust the new CA everywhere before anything presents it

The trust bundle is just concatenated PEMs, so trusting two issuers costs one cat. Deploy it to every terminator — every region, every canary, every stack somebody forgot — and only then continue.

# Trust bundle: both issuers, old first for readability.
cat ca-a.crt ca-b.crt > /etc/nginx/certs/client-ca-bundle.pem

# Sanity-check what is actually in the bundle before reloading.
openssl crl2pkcs7 -nocrl -certfile /etc/nginx/certs/client-ca-bundle.pem \
  | openssl pkcs7 -print_certs -noout

server {
    listen 443 ssl;
    server_name hooks.partner.example;
    ssl_protocols TLSv1.3;

    # Both issuers are acceptable for the duration of the rotation.
    ssl_client_certificate /etc/nginx/certs/client-ca-bundle.pem;
    ssl_verify_client on;
    ssl_verify_depth  2;

    location /webhooks/payments {
        if ($ssl_client_verify != SUCCESS) { return 403; }
        proxy_set_header X-Client-Verify  $ssl_client_verify;
        proxy_set_header X-Client-Subject $ssl_client_s_dn;
        # Issuer DN lets you prove when the old CA stops being used.
        proxy_set_header X-Client-Issuer  $ssl_client_i_dn;
        proxy_pass http://127.0.0.1:8080;
    }
}

nginx -t && nginx -s reload applies the bundle without dropping established connections. Forwarding $ssl_client_i_dn is what makes Step 4 a measurement rather than a guess: the application can now count handshakes per issuer.

Step 3: Distribute the new key pair and hot-reload senders

Push the new certificate and key into the secret store as a new version, let each sender read it from disk or from the store’s SDK, and swap the TLS identity in-process. Restarting the fleet works too, but a rolling restart drops keep-alive connections and any in-flight delivery on them; an in-process swap does not.

The sender keeps its TLS identity behind a single mutable agent reference. Building a new agent and closing the old one lets in-flight requests finish on the old sockets while every new request uses the new identity.

import { readFile } from 'node:fs/promises';
import { watch } from 'node:fs';
import { X509Certificate } from 'node:crypto';
import { Agent, request } from 'undici';

const CERT_PATH = process.env.WEBHOOK_CLIENT_CERT ?? '/etc/webhook/tls/client-fullchain.pem';
const KEY_PATH = process.env.WEBHOOK_CLIENT_KEY ?? '/etc/webhook/tls/client.key';
const SERVER_CA = process.env.WEBHOOK_SERVER_CA ?? '/etc/webhook/tls/server-ca.pem';

let identity: Agent | null = null;

async function buildAgent(): Promise<Agent> {
  const [cert, key, ca] = await Promise.all([readFile(CERT_PATH), readFile(KEY_PATH), readFile(SERVER_CA)]);
  // X509Certificate reads the FIRST certificate in the PEM: the leaf, if the chain is ordered correctly.
  const leaf = new X509Certificate(cert);
  const notAfter = new Date(leaf.validTo);
  const notBefore = new Date(leaf.validFrom);
  const now = new Date();
  if (now < notBefore || now >= notAfter) {
    throw new Error(`refusing to load client cert outside validity: ${leaf.validFrom} .. ${leaf.validTo}`);
  }
  return new Agent({
    connect: { cert, key, ca, minVersion: 'TLSv1.3' },
    keepAliveTimeout: 30_000,
    keepAliveMaxTimeout: 60_000,
  });
}

export async function reloadClientIdentity(): Promise<void> {
  const next = await buildAgent();
  const previous = identity;
  identity = next;               // new deliveries use the new leaf immediately
  await previous?.close();       // graceful: in-flight requests drain on old sockets
  const leaf = new X509Certificate(await readFile(CERT_PATH));
  console.info({ event: 'mtls_identity_reloaded', subject: leaf.subject, not_after: leaf.validTo });
}

export async function deliver(url: string, body: string): Promise<number> {
  if (!identity) await reloadClientIdentity();
  const res = await request(url, {
    method: 'POST',
    body,
    headers: { 'content-type': 'application/json' },
    dispatcher: identity!,
  });
  await res.body.dump();
  return res.statusCode;
}

// Trigger a reload when the mounted secret changes, and on SIGHUP for manual control.
watch(CERT_PATH, { persistent: false }, () => {
  reloadClientIdentity().catch((err) => console.error({ event: 'mtls_reload_failed', err: String(err) }));
});
process.on('SIGHUP', () => {
  reloadClientIdentity().catch((err) => console.error({ event: 'mtls_reload_failed', err: String(err) }));
});

Roll this out to one sender first and confirm the receiver logs a SUCCESS verify with the new issuer DN before the rest of the fleet follows. Because both leaves are valid and both CAs are trusted, a partially rolled-out fleet is a supported state, not a race.

Step 4: Retire the old certificate and the old CA

Retirement is a separate change, made only after evidence. Count handshakes by issuer on the receiving side and wait for the old issuer to go to zero and stay there for at least one full retry cycle — a sender that only fires on month-end batches will otherwise reappear after you have deleted its trust anchor.

import type { Request, Response, NextFunction } from 'express';
import { Counter } from 'prom-client';

export const handshakesByIssuer = new Counter({
  name: 'webhook_mtls_handshakes_total',
  help: 'Verified inbound mTLS handshakes, labelled by issuing CA',
  labelNames: ['issuer', 'subject'] as const,
});

export function recordIssuer(req: Request, res: Response, next: NextFunction) {
  if (req.header('x-client-verify') !== 'SUCCESS') return res.status(403).end();
  const issuer = req.header('x-client-issuer') ?? 'unknown';
  handshakesByIssuer.labels(issuer, req.header('x-client-subject') ?? 'unknown').inc();
  next();
}

When sum(increase(webhook_mtls_handshakes_total{issuer=~".*CA A.*"}[7d])) is 0, drop ca-a.crt from the bundle, reload, and destroy the old private key in the secret store (delete the version; do not merely overwrite the file on one host). Only now is the rotation finished.

Step 5: Automate renewal and alert on approaching expiry

Rotation you perform by hand once a quarter will eventually be skipped. Export days-until-expiry for every certificate the sender loads and for every CA in the bundle, so the alert fires with enough lead time to run the whole overlap procedure rather than a panicked swap.

import { readFile } from 'node:fs/promises';
import { X509Certificate } from 'node:crypto';
import { Gauge } from 'prom-client';

export const certExpiryDays = new Gauge({
  name: 'tls_certificate_expiry_days',
  help: 'Days until notAfter for a TLS certificate in use',
  labelNames: ['role', 'subject'] as const,
});

export function daysUntilExpiry(pem: string | Buffer, now = new Date()): number {
  const cert = new X509Certificate(pem);
  return (new Date(cert.validTo).getTime() - now.getTime()) / 86_400_000;
}

/** Split a bundle into individual PEM blocks so every CA is measured, not just the first. */
export function splitPemBundle(bundle: string): string[] {
  return bundle.match(/-----BEGIN CERTIFICATE-----[\s\S]*?-----END CERTIFICATE-----/g) ?? [];
}

export async function sampleExpiry(paths: Record<string, string>): Promise<void> {
  for (const [role, path] of Object.entries(paths)) {
    const pem = await readFile(path, 'utf8');
    for (const block of splitPemBundle(pem)) {
      const cert = new X509Certificate(block);
      certExpiryDays.labels(role, cert.subject).set(daysUntilExpiry(block));
    }
  }
}

setInterval(() => {
  sampleExpiry({
    client_leaf: process.env.WEBHOOK_CLIENT_CERT!,
    trust_bundle: process.env.WEBHOOK_SERVER_CA!,
  }).catch((err) => console.error({ event: 'expiry_sample_failed', err: String(err) }));
}, 3_600_000).unref();

Alert at 21 days as a warning and 7 days as a page: min(tls_certificate_expiry_days) by (role, subject) < 21. Twenty-one days is not arbitrary — it is one trust rollout, one canary, one soak, and one retirement with slack. If your issuance is automated through an ACME-style or cert-manager flow, keep the alert anyway: the failure you are guarding against is the automation silently stopping.

Verification and testing

Prove both certificates work simultaneously during the overlap, which is the property the whole procedure depends on:

# Old leaf must still be accepted during the overlap window.
openssl s_client -connect hooks.partner.example:443 -tls1_3 \
  -cert client-a.crt -key client-a.key -CAfile server-ca.pem </dev/null 2>&1 \
  | grep -E 'Verify return code|Verification'

# New leaf must be accepted too, before any sender is cut over.
openssl s_client -connect hooks.partner.example:443 -tls1_3 \
  -cert client-b-fullchain.pem -key client-b.key -CAfile server-ca.pem </dev/null 2>&1 \
  | grep -E 'Verify return code|Verification'

# End-to-end with the new identity: expect 204, not 400 "No required SSL certificate".
curl -sS -o /dev/null -w '%{http_code}\n' https://hooks.partner.example/webhooks/payments \
  --cert client-b-fullchain.pem --key client-b.key --cacert server-ca.pem \
  -H 'Content-Type: application/json' -d '{"event":"payment.succeeded"}'

# Key and certificate must belong together; these two hashes must match.
openssl x509 -noout -modulus -in client-b.crt | openssl md5
openssl rsa  -noout -modulus -in client-b.key | openssl md5

Guard the invariants in CI so a bad bundle never ships:

import { describe, expect, it } from 'vitest';
import { readFileSync } from 'node:fs';
import { X509Certificate } from 'node:crypto';
import { daysUntilExpiry, splitPemBundle } from './expiry.js';

const bundle = readFileSync('deploy/certs/client-ca-bundle.pem', 'utf8');
const leaf = readFileSync('deploy/certs/client-fullchain.pem', 'utf8');

describe('mTLS rotation invariants', () => {
  it('trusts both issuers during the overlap', () => {
    expect(splitPemBundle(bundle)).toHaveLength(2);
  });

  it('keeps at least 21 days of leaf validity', () => {
    expect(daysUntilExpiry(leaf)).toBeGreaterThan(21);
  });

  it('ships the leaf first in the chain file', () => {
    const first = new X509Certificate(splitPemBundle(leaf)[0]);
    expect(first.subject).toContain('CN=payments-provider');
    expect(first.ca).toBe(false);
  });

  it('has an issuer that is present in the trust bundle', () => {
    const first = new X509Certificate(splitPemBundle(leaf)[0]);
    const issuers = splitPemBundle(bundle).map((pem) => new X509Certificate(pem).subject);
    expect(issuers).toContain(first.issuer);
  });
});

Failure modes and gotchas

• Presenting before trusting. Deploy the new leaf to senders while a single terminator still has only the old CA and that terminator answers every handshake with unknown ca — a TLS alert, not an HTTP status, so your application dashboards stay green while deliveries fail. Always complete the trust rollout, verify it on each region with openssl s_client, and only then start the sender rollout.
• Keep-alive connections mask the cutover. Long-lived pooled sockets keep using the identity they were established with, so a sender can report success for hours on the old certificate and then fail the moment the pool churns — often after you have retired the old CA. Closing the previous agent after the swap drains those sockets deliberately; the same connection-lifetime reasoning applies to connection pooling and keep-alive for webhook senders.
• The chain, not the leaf, is what fails verification. If the new leaf is signed by an intermediate and you ship only the leaf, the terminator cannot build a path and rejects a perfectly valid certificate. Concatenate leaf then intermediate into the file the client presents, keep the root in the receiver’s bundle, and raise ssl_verify_depth to cover the extra hop.
• Rotating the certificate but reusing the key. Re-signing the old public key produces a new serial and a new expiry while leaving the exact secret you were trying to retire in place. Generate a new key every rotation, and delete the previous version in the secret store at Step 4 so a rollback cannot silently resurrect it — the same discipline applied to shared secrets in how to implement secure key rotation for webhooks.

Frequently Asked Questions

Related

• Configuring mTLS for webhook endpoints — the terminator and application configuration this rotation operates on.
• Mutual TLS for Webhooks — the trust model, pinning, and lifecycle concepts behind client certificates.
• Zero-downtime webhook secret rotation — the same overlap pattern applied to shared HMAC secrets.
• Storing webhook secrets in a secrets manager — how the new key reaches the fleet without touching a build artefact.