Storing webhook secrets in a secrets manager

WEBHOOK_SECRET in the environment works right up until you have four hundred tenants, an auditor asking who read which secret and when, and a rotation that requires a redeploy of every consumer. Environment variables have no version history, no per-read audit trail, no access control finer than “whoever can read the process”, and they leak into crash dumps, container inspect output, and CI logs. This page covers moving webhook signing secrets into AWS Secrets Manager or HashiCorp Vault without paying an API call per delivery — the constraint that makes this harder than it first looks. It extends the Key Rotation Strategies reference, and it is the storage layer underneath the procedure in Zero-downtime webhook secret rotation: once secrets live in a managed store, the dual-secret overlap becomes a property of the stored document rather than a comma-separated environment variable.

The tension to resolve is latency against freshness. A high-volume webhook endpoint verifies thousands of signatures per second, and Secrets Manager is a rate-limited, network-round-trip API billed per call — you cannot fetch on every request. But a cached secret is a stale secret, and staleness during a rotation is exactly when it hurts. Everything below is a way of making the cache both fast and provably bounded in how wrong it can be.

Prerequisites

• Runtime: Node.js 20+ with TypeScript 5+ and @aws-sdk/client-secrets-manager 3.x (npm i @aws-sdk/client-secrets-manager). The Vault equivalents are noted where they differ.
• A managed store: AWS Secrets Manager, or Vault with a KV v2 mount that supports versioned reads.
• Workload identity, not static credentials — an IRSA-annotated service account, an ECS task role, or a Vault Kubernetes auth role. Storing a secret in a manager that you unlock with a long-lived key in an environment variable moves the problem, it does not solve it.
• An existing HMAC verifier to feed, such as the one in Step-by-step HMAC webhook validation in Node.js.
• A defined rotation cadence and an owner for it. The store makes rotation cheap; it does not make it happen. Pair this with the runbook in How to implement secure key rotation for webhooks.

Step 1: Model the secret and choose a per-tenant naming scheme

Store a small JSON document rather than a bare string. A document lets one secret carry the overlap window — the current material and the outgoing one — so the verifier gets everything it needs from a single read, and rotation becomes an update to one object rather than a coordinated edit across several.

export interface SecretMaterial {
  readonly id: string;          // stable label, e.g. "2026-07-25"
  readonly value: string;       // the HMAC key, hex-encoded
  readonly activeFrom: string;  // ISO-8601
}

export interface WebhookSecretDocument {
  readonly version: 1;
  /** Newest first. Length 1 normally, 2 during an overlap window. */
  readonly materials: readonly SecretMaterial[];
}

const TENANT_PATTERN = /^[a-z0-9][a-z0-9-]{2,62}$/;

/** Deterministic, hierarchical, and safe to build from untrusted input. */
export function secretName(env: string, provider: string, tenantId: string): string {
  if (!TENANT_PATTERN.test(tenantId) || !TENANT_PATTERN.test(provider)) {
    throw new Error('ERR_SECRET_NAME_INVALID');
  }
  return `webhook/${env}/${provider}/${tenantId}`;
}

The TENANT_PATTERN guard is not cosmetic. Tenant identifiers usually arrive from a request path or a header, and secret names are path-like in both Secrets Manager and Vault. An unvalidated identifier containing ../ or a wildcard turns a lookup into a probe of your entire secret hierarchy, limited only by whatever your IAM policy forgot to exclude.

Engineering Note: Put the environment first in the name. webhook/prod/... and webhook/staging/... are then separable by a single ARN prefix in IAM, which is what makes the least-privilege policy in Step 4 expressible in one line instead of an enumeration.

Step 2: Cache the secret in-process behind a bounded TTL

Secrets Manager GetSecretValue is quota-limited and adds tens of milliseconds to a request that should complete in single-digit milliseconds. Cache it. The cache needs three properties that a plain Map does not give you: a TTL, a size cap so a tenant enumeration attack cannot exhaust memory, and single-flight behaviour so a cold key under load produces one API call rather than five hundred.

import {
  SecretsManagerClient,
  GetSecretValueCommand,
  ResourceNotFoundException,
} from '@aws-sdk/client-secrets-manager';

interface Entry {
  readonly document: WebhookSecretDocument;
  readonly expiresAt: number;
}

export class SecretCache {
  private readonly entries = new Map<string, Entry>();
  private readonly inflight = new Map<string, Promise<WebhookSecretDocument>>();

  constructor(
    private readonly client: SecretsManagerClient,
    private readonly ttlMs = 300_000,
    private readonly maxEntries = 5_000,
  ) {}

  async get(name: string): Promise<WebhookSecretDocument> {
    const hit = this.entries.get(name);
    if (hit && hit.expiresAt > Date.now()) {
      // Refresh recency for the LRU bound.
      this.entries.delete(name);
      this.entries.set(name, hit);
      return hit.document;
    }

    // Single flight: concurrent misses on the same name share one API call.
    const existing = this.inflight.get(name);
    if (existing) return existing;

    const pending = this.fetch(name).finally(() => this.inflight.delete(name));
    this.inflight.set(name, pending);
    return pending;
  }

  invalidate(name: string): void {
    this.entries.delete(name);
  }

  private async fetch(name: string): Promise<WebhookSecretDocument> {
    let raw: string;
    try {
      const res = await this.client.send(
        new GetSecretValueCommand({ SecretId: name, VersionStage: 'AWSCURRENT' }),
      );
      raw = res.SecretString ?? '';
    } catch (err) {
      if (err instanceof ResourceNotFoundException) throw new Error('ERR_SECRET_NOT_FOUND');
      throw new Error('ERR_SECRET_STORE_UNAVAILABLE');
    }

    const document = JSON.parse(raw) as WebhookSecretDocument;
    if (document.version !== 1 || document.materials.length === 0) {
      throw new Error('ERR_SECRET_DOCUMENT_INVALID');
    }

    // Evict the least-recently-used entry once the cap is reached.
    if (this.entries.size >= this.maxEntries) {
      const oldest = this.entries.keys().next().value;
      if (oldest !== undefined) this.entries.delete(oldest);
    }
    this.entries.set(name, { document, expiresAt: Date.now() + this.ttlMs });
    return document;
  }
}

Note that failures are never cached. A transient ERR_SECRET_STORE_UNAVAILABLE that landed in the map would keep rejecting deliveries for a full TTL after the store recovered. Only successful reads populate an entry.

Wiring it into verification is then a two-line change from an environment variable read, and the overlap window comes along for free:

import crypto from 'node:crypto';

const cache = new SecretCache(new SecretsManagerClient({}));

export async function verifyDelivery(
  tenantId: string,
  rawBody: Buffer,
  signatureHex: string,
): Promise<{ ok: boolean; materialId?: string }> {
  const doc = await cache.get(secretName(process.env.APP_ENV!, 'acme', tenantId));
  const provided = Buffer.from(signatureHex, 'hex');

  let matched: string | undefined;
  for (const material of doc.materials) {
    const expected = crypto
      .createHmac('sha256', Buffer.from(material.value, 'hex'))
      .update(rawBody)
      .digest();
    // Evaluate every material so timing does not reveal which one matched.
    if (expected.length === provided.length && crypto.timingSafeEqual(expected, provided)) {
      matched = material.id;
    }
  }
  return { ok: matched !== undefined, materialId: matched };
}

Emit materialId as a metric label. When it reads 2026-07-25 for a full TTL and never the previous label, the outgoing material is provably unused and safe to drop from the document.

Step 3: Invalidate the cache the moment a rotation lands

A TTL alone means a rotation takes up to five minutes to reach every process, and a much longer TTL — which you want, for cost and latency — makes that worse. Push invalidation on top of the TTL: Secrets Manager emits a rotation event to EventBridge, and Vault can fire a webhook from a rotation job. Treat the push as an optimisation and the TTL as the guarantee, because a missed event must not be able to strand a process on old material forever.

import type { Handler } from 'aws-lambda';

/**
 * EventBridge rule:
 *   source = ["aws.secretsmanager"]
 *   detail-type = ["AWS Secrets Manager Rotation Succeeded"]
 * Fan this out to every verifier instance (SNS topic, Redis pub/sub, or a
 * control-plane endpoint) and call cache.invalidate on each.
 */
interface RotationEvent {
  readonly detail: { readonly additionalEventData?: { readonly SecretId?: string } };
}

export const onRotation: Handler<RotationEvent> = async (event) => {
  const arn = event.detail.additionalEventData?.SecretId;
  if (!arn) return;
  // ARN tail is "webhook/prod/acme/tenant-42-AbCdEf"; strip the 6-char suffix.
  const name = arn.split(':secret:')[1]?.replace(/-[A-Za-z0-9]{6}$/, '');
  if (!name?.startsWith('webhook/')) return;
  await broadcastInvalidation(name);
};

// Receiving side, inside each verifier process.
export function subscribeToInvalidations(cache: SecretCache, bus: InvalidationBus): void {
  bus.on('secret.rotated', (name: string) => {
    cache.invalidate(name);
    // Do NOT prefetch here. A rotation across thousands of tenants would
    // stampede the store; let the next real delivery repopulate lazily.
  });
}

Refusing to prefetch on invalidation is the important line. A provider-wide rotation invalidates every tenant entry at once, and an eager refill turns that into thousands of simultaneous GetSecretValue calls — throttling, then failures, then a self-inflicted outage during a routine security operation. Lazy repopulation spreads the same work across the arrival rate of real traffic. If you do want warm caches, add jitter to the TTL itself so entries expire on a spread rather than in lockstep, the same reasoning behind adding jitter to webhook retry backoff.

Step 4: Scope the IAM policy to a single read action

The verifier reads. It never writes, rotates, deletes, tags, or lists. Every one of those verbs on the verifier’s role is a way for a compromised request handler to escalate from reading one tenant’s secret to overwriting all of them — and an overwritten signing secret is a silent, total authentication bypass, because the attacker then controls what a valid signature looks like.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadWebhookSigningSecrets",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:webhook/prod/*",
      "Condition": {
        "StringEquals": { "aws:PrincipalTag/svc": "webhook-verifier" }
      }
    },
    {
      "Sid": "DecryptWithTheWebhookKeyOnly",
      "Effect": "Allow",
      "Action": "kms:Decrypt",
      "Resource": "arn:aws:kms:eu-west-1:123456789012:key/8f2c1b60-0000-4a11-9f00-webhookcmk1",
      "Condition": {
        "StringEquals": { "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com" }
      }
    }
  ]
}

The second statement is the one teams forget. A customer-managed KMS key means GetSecretValue also needs kms:Decrypt, and the kms:ViaService condition ensures that grant only works through Secrets Manager — the role cannot use the key to decrypt anything else. The Vault equivalent is a policy with capabilities = ["read"] on secret/data/webhook/prod/* and no create, update, delete, or list; omitting list matters, because a role that can enumerate the path learns your full tenant roster even if it cannot read a single value.

Verification and testing

• Cache hit ratio. Wrap the store client in a counting stub and assert that 1,000 verifications for one tenant produce exactly one GetSecretValue call. This is the test that catches a refactor accidentally constructing a new SecretCache per request — the most common way this optimisation silently disappears.
• Single-flight under concurrency. Fire 50 concurrent misses for the same name at a client stubbed with a 50 ms delay; assert exactly one call reached the store and all 50 promises resolved with the same document.
• Failures are not cached. Make the first fetch throw, then succeed; assert the second call reaches the store rather than returning a cached error, and that the successful document is then cached.
• Invalidation latency. Rotate a secret in a staging account and assert that a delivery signed with the new material verifies within seconds, not within a TTL. Measure it, and alert if it regresses past your rotation runbook’s assumption.
• IAM negative test. Assert the verifier role is denied PutSecretValue and denied reads outside its prefix, using the policy simulator rather than inspecting the policy document by eye.

import { SecretCache } from './secret-cache';

class CountingClient {
  public calls = 0;
  async send(): Promise<{ SecretString: string }> {
    this.calls += 1;
    await new Promise((r) => setTimeout(r, 50));
    return {
      SecretString: JSON.stringify({
        version: 1,
        materials: [{ id: '2026-07-25', value: 'aabb', activeFrom: '2026-07-25T00:00:00Z' }],
      }),
    };
  }
}

test('concurrent misses collapse into a single store call', async () => {
  const client = new CountingClient();
  const cache = new SecretCache(client as never, 300_000, 100);
  const results = await Promise.all(
    Array.from({ length: 50 }, () => cache.get('webhook/prod/acme/tenant-42')),
  );
  expect(client.calls).toBe(1);
  expect(new Set(results.map((r) => r.materials[0].id)).size).toBe(1);
});

test('a warm entry never touches the store again', async () => {
  const client = new CountingClient();
  const cache = new SecretCache(client as never, 300_000, 100);
  for (let i = 0; i < 1_000; i += 1) await cache.get('webhook/prod/acme/tenant-42');
  expect(client.calls).toBe(1);
});

# Prove the role cannot write, only read, before you ship it.
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/webhook-verifier \
  --action-names secretsmanager:GetSecretValue secretsmanager:PutSecretValue \
  --resource-arns arn:aws:secretsmanager:eu-west-1:123456789012:secret:webhook/prod/acme/tenant-42-AbCdEf \
  --query 'EvaluationResults[].{action:EvalActionName,decision:EvalDecision}' --output table
# Expect: GetSecretValue allowed, PutSecretValue implicitDeny

Failure modes and gotchas

• Calling the store on every request. It works perfectly in staging at two deliveries per minute and collapses in production: GetSecretValue throttles, every retry adds latency, and the bill grows linearly with webhook volume. Worse, the throttled path usually fails open somewhere, because nobody wants deliveries dropped by a quota error. Cache first, and treat a rising GetSecretValue call rate per delivery as a regression alarm.
• An unbounded cache keyed by tenant. A Map with no size cap and no eviction is a memory leak addressed by tenant identifier. An attacker who can trigger lookups for arbitrary tenant ids — or a bug that builds a slightly different name each call — grows it without limit until the process is killed mid-delivery. The size cap in SecretCache is a denial-of-service control, not a tidiness measure.
• Failing open when the store is unreachable. Under pressure, “accept the delivery, we can verify later” is a tempting patch. It is an unauthenticated endpoint for the duration of the incident, and there is no later. Fail closed with 503 so the sender retries, and let the retry budget cover a short outage — see choosing retry budgets and max attempts for sizing that window against your grace period.
• Leaking the secret through error handling. Constructing an exception message that interpolates the fetched document, attaching the parsed secret to a span attribute, or letting an unhandled rejection print the AWS SDK response object all put key material into logs that are retained far longer than the key is. Wrap the value in a type whose toString and toJSON return a redaction marker, and add a test that asserts JSON.stringify(material) contains no hex.
• Granting secretsmanager:* “just for now”. The wildcard added during setup to unblock a deploy is never removed, and it converts a read-only verifier into a principal that can silently replace every tenant’s signing key. Start from the two statements above and add only what a failing call proves you need, verified with the policy simulator rather than by trying it in production.

Frequently Asked Questions

Related

• Zero-downtime webhook secret rotation
• How to implement secure key rotation for webhooks
• Key Rotation Strategies
• Rotating JWT signing keys with JWKS
• HMAC Signature Verification