Cache Breakpoint Lookback Exceeds 20 Blocks

The mistake

Assuming that a prefix cached earlier in a long, busy conversation will always be found on a later read. The read-side lookup doesn't search the conversation's whole history — it checks a bounded set of candidate boundaries in the incoming request, and a prefix ending anywhere else is invisible to it.

Why this happens

OpenAI's caching guide now spells out exactly which boundaries a GPT-5.6-or-later request is checked against. In explicit-only mode: the first 2 and the latest 50 explicit breakpoints. In implicit mode: those same explicit breakpoints, plus the implicit breakpoint, up to 20 earlier eligible message endings, and the end of the opening block of developer messages. As a conversation grows, older message endings fall out of that window — not because anything about them changed, but because newer ones pushed in. Developer messages after the opening block aren't automatic lookup boundaries at all.

Why it matters

A workload that relies on implicit caching to rediscover an old prefix sees its hit rate fall as conversations lengthen, even though every prefix cached correctly when it was written. Spot-checking a single request won't show it: the problem only appears deep into a thread, and it looks like a flaky cache rather than a bounded window doing exactly what it is specified to do.

The fix

Put the content you want reused for a whole session at the start, and mark it: the first explicit breakpoints are always checked, however long the conversation gets. Add explicit breakpoints after stable content you expect to reuse rather than relying on implicit message endings, and if you edit a message mid-thread, append a new one instead — extending a message moves its ending inside it, which stops the old prefix matching. Prompt Cache Diagnostics, on the Responses API, names the reason for a specific miss.

Recognizing it

The signature is a hit rate that's healthy early in a conversation and decays as the thread lengthens, while new conversations still cache well. Prompt Cache Diagnostics, comparing a late request with an earlier response, will name the miss, and the Prompt Caching dashboard shows the decline as a trend rather than a scatter of individual misses.