Prevent stampedes

A cache is a load absorber, and a stampede is the moment it stops absorbing. This page covers the concept, the mechanism that already handles it, and the automatic Redis coordination that applies when one process is not the whole picture.

The thundering herd

The thundering herd is a general system design failure, not a caching quirk. A single event wakes many waiters at once, all of them do the same work, and the resource they contend for collapses under a load spike it never sees during normal operation.

In a cache it looks like this. A popular key expires, a deploy starts servers with cold L1 caches, or a new key suddenly becomes hot. The next second brings 10,000 requests for it, every one of them misses, and every one of them calls your database with the identical query. Your steady state was one query per key per TTL. Your worst second is 10,000 queries, and it arrives precisely when the key is hottest.

The damage is not linear. Those 10,000 queries take connections from a bounded pool, so unrelated requests queue behind them, and the slower the origin gets the longer each caller holds its slot. That is a feedback loop, which is why a stampede tends to show up as a full outage rather than a latency bump.

In-flight dedupe

This is on by default. You are not switching it on, and you should not switch it off.

The first caller for a key stores its promise. Every caller that arrives while that promise is unsettled receives the same promise instead of starting a second load. When it settles, all of them resolve from one result and the entry is removed.

import { cache } from '../cache/index.js'
import { db } from '../db.js'

export async function getProduct(id) {
  return cache.getOrSet(`product:${id}`, async ({ signal } = {}) => {
    // Runs once per cold key, no matter how many callers are waiting.
    return db.products.findById(id, { signal })
  })
}

Nothing in that code asks for dedupe. The collapse is a property of getOrSet.

What the benchmark measured

benchmarks/herd.mjs fires 10,000 concurrent getOrSet calls at one cold key, with a loader that sleeps 25 ms to stand in for a database round trip, and counts how many times the loader actually runs.

Both runs assert that every caller received the correct value, so the collapse cannot come from dropping requests. Reproduce it yourself:

npm run build
NODE_ENV=production node benchmarks/herd.mjs

The second row is there as a control, not as a configuration you should copy. It exists so the first row means something.

Tuning the dedupe window

Tuning here means numbers. The two that matter are how long an in-flight entry stays reusable, and how many distinct keys can be tracked at once.

export const cache = new LazyLayersCache({
  ttlMs: 5 * 60 * 1000,

  inflight: {
    // The dedupe lifetime adapts to the wait and loader budgets automatically.
    // How many distinct keys can be tracked at once. Size it to your concurrent
    // cold key count, not to your total keyspace.
    maxEntries: 10_000,
  },
})

inflight.ttlMs is derived automatically from the loader hard timeout, the distributed wait budget when locking is active, and a 50 ms margin, with a 5,000 ms minimum. Default settings produce 10,050 ms without locking and 20,100 ms with Redis locking. An explicit value still overrides this. Setting it below the time a healthy load needs can allow duplicate work. Settled promises are removed immediately.

maxEntries caps the tracking map. Past the cap, a new load still runs and still returns the right value, it simply does not register for reuse, and it emits inflight:bypass with reason maxEntries. A steady stream of those means your cold key concurrency outgrew the cap.

When one process's dedupe is not enough

In-flight dedupe is per process. Four servers each collapse their own callers to one loader call, which is four queries rather than one. Usually that is fine. It stops being fine when the loader is genuinely expensive, when it hits a rate limited third party, or when you have twenty servers rather than four.

With Redis L2, getOrSet automatically coordinates concurrent misses across instances, including misses after cache expiry. No lock configuration is required. If only L1 expires, a live L2 value refills it without invoking the loader.

import { LazyLayersCache, RedisStore } from 'lazy-layers-cache'
import { redis } from './redis.js'
import { bus } from './bus.js'

export const cache = new LazyLayersCache({
  l2: new RedisStore(redis, { prefix: 'app:cache:' }),
  eventBus: bus,
  source: process.env.INSTANCE_ID ?? `srv-${process.pid}`,

  // Locking and renewal are automatic with Redis L2.
})

What happens on a cold or expired key

The default wait budget is derived from the longer of the lock TTL and the loader hard timeout, plus one poll interval. With default settings that is 10,050 ms. This allows a healthy slow loader to finish while keeping waiting bounded. Explicit waitTimeoutMs settings still take precedence.

Lock defaults

RedisStore renews with a token-checked Lua operation at roughly one third of the lease lifetime. A failed renewal or expired local lease aborts the loader signal and rejects with DistributedLockLostError, or returns eligible stale data. Results that arrive after detected ownership loss are discarded. Custom stores can implement renewLock(key, token, ttlMs) to support renewal. Without it, their loads must finish within the lease.

This is cache stampede protection, not an exactly-once execution guarantee. Redis failure before contention is observed still uses the existing fail-open origin path. Redis failover, process pauses and loaders that ignore their abort signal can allow overlapping origin work. Pass the supplied signal to clients that support cancellation. The lease and wait budget do not replace Redis client command timeouts.

Choosing

There is a third mechanism working alongside these, and it is easy to miss. When a getOrSet loader resolves with an event bus configured, the value itself is published, and every peer writes it straight into its own L1. One server pays for the query and the rest are warm before they ever ask, which flattens the herd before it forms.

Watching the collapse

Every reuse emits an event, so the collapse is measurable rather than assumed.

import { cache } from './index.js'

cache.on((event) => {
  // The herd collapsing. High counts next to low loader:start counts is health.
  if (event.type === 'inflight:reuse') metrics.increment('cache.inflight.reuse')
  // Tracking capacity exceeded. A steady stream means raise inflight.maxEntries.
  if (event.type === 'inflight:bypass') metrics.increment('cache.inflight.bypass')
  if (event.type === 'loader:start') metrics.increment('cache.loader.start')
})

Where to next