Cache Breakpoint Lookback Exceeds 20 Blocks

The mistake

Assuming that because a prefix was cached at some point earlier in a long-running, high-throughput session, it will always be found on a later read — and not noticing that the read-side lookback only considers a bounded number of the most recent breakpoints, not the full history of everything ever written.

Why this happens

Writes and reads aren't symmetric in how many breakpoints they touch. A single request can only register a small number of new writes, but matching a read against prior breakpoints looks back across a much larger set — still bounded, not infinite. On a session or service that's cycling through many distinct prefixes quickly, older breakpoints age out of that lookback window as newer ones push in, purely from volume, with nothing about the individual prefix itself changing.

Why it matters

A workload assuming unlimited lookback will see a cache hit rate that degrades specifically under high breakpoint churn — lots of distinct prefixes in flight — even though any individual prefix, tested in isolation, would have cached correctly. That's a hard pattern to notice from spot-checking single requests, because the problem only appears at volume, and it looks identical to a flaky cache rather than a bounded window doing exactly what it's specified to do.

The fix

Treat the read-side lookback as a real capacity limit, not a formality, on any workload registering a large number of distinct cacheable prefixes in a short window. If a specific prefix genuinely needs to stay reliably cacheable across a long gap or high churn from other traffic, that's exactly the case the extended-retention path — offloading cached state off the default in-memory path — is meant to cover, and it's worth checking whether your workload qualifies rather than assuming the default lookback will always find what you wrote earlier.

A useful mental model: think of the lookback as a short queue, not an archive. Every new breakpoint written by any traffic sharing the same matching pool pushes the queue forward, and whatever falls off the back stops being a candidate for a read match — regardless of how useful or recently-relevant that particular prefix still is to your own workload specifically.

See also

Prompt Caching, Fully Specified lists both the write cap and the read lookback cap side by side, since confusing the two is the most common way this gets misdiagnosed.