Prompt caching: what breaks it, what it saves, and what we measured

What prompt caching is

A model handles a request in two phases. First it reads the whole prompt (the prefill) and builds an internal representation of every token, called the key-value (KV) cache. Then it generates the answer one token at a time (the decode). Prefill is the part you pay input prices for, and for long prompts it is a large share of the time before the first token appears.

If two requests start with the same tokens, the representation of those tokens is identical too, because each token's representation depends only on the tokens before it. Prompt caching keeps that work for a few minutes and reuses it when another request starts the same way. The model still sees the full prompt, so answers don't change. Only the price and the prefill time do.

It's easy to confuse with two other kinds of caching:

Table 1. Prompt caching is the only one of the three that never changes what the model says. That makes it safe to turn on everywhere.

The one rule: prefix match

Everything in this article follows from one rule: the cache matches from the first token forward, and stops at the first difference. It doesn't look for the longest matching chunk somewhere in the middle. If request 2 differs from request 1 at token 50, only the first 49 tokens can be reused, even if the next 5,000 are identical.

Two consequences matter in practice:

• Order your prompt from most stable to least stable. Content that never changes goes first, then content that is fixed for one user or conversation, then content that changes on every request.
• Know the order the provider uses. Your API call has separate fields, but the provider joins them into one sequence. Claude's documented order is tools, then system, then messages, so a change in the tool list invalidates everything. OpenAI says the cached prefix includes "tool definitions" and Structured Outputs schemas. Gemini doesn't document its order. Treat tools and system instructions as the front of the prompt.

How each provider does it

All three providers cache prefixes, but they differ in whether you have to ask, what a write costs, and how long an entry lives. Prices below are list prices per million input tokens, checked on 7 October 2026.

Table 2. Sources: Claude prompt caching and pricing; Gemini caching, explicit caching and pricing (3.8 Flash prices are promotional through 31 December 2026); OpenAI prompt caching and pricing. Prices and minimums change, so check them before you rely on them.

Where a write costs extra, caching pays off only if the entry is read again before it expires. On Claude, the docs work out that a 5-minute entry breaks even after one read, since 1.25× + 0.1× is less than 2×, and a 1-hour entry after two reads. For Gemini's explicit cache, here is our own calculation from the prices above. Each cached read of our 5,384-token prefix saves $0.0036, and storing it costs $0.0027 an hour. So it pays off from about one request per hour.

Eight ways to break the cache

None of these produce an error. The request succeeds, the answer is fine, and the bill is higher. Each one has a matching example in the playground.

Current time: 2026-10-07T14:32:09Z
You are the support assistant…
[4,000-token handbook]

You are the support assistant…
[4,000-token handbook]
…
User: Today's date: 7 October 2026
Where is my order?

The experiment

Docs describe the rules. We wanted to see them in real requests: how much of a realistic prompt each layout leaves reusable, whether the provider actually caches it, and what that does to cost and latency. We had Gemini API access, so we ran on Gemini 3.8 Flash (gemini-3.8-flash), on 7 October 2026.

The prompt

A customer-support assistant for Northwind Outfitters, a fictional outdoor-gear store. Every layout sends the same parts:

Table 3. Gemini counted about 5,400 prompt tokens per request, comfortably above the 4,096-token minimum. The full text is in src/fixture.js.

Eight layouts

Table 4. Each layout starts its system instruction with a tag that is fixed within the layout, such as deployment r202610079c02-stable. It stops layouts from reading each other's cache, the way two separate applications wouldn't share one.

Procedure and measures

• 12 requests per layout, one at a time, 1.5 seconds apart, layout after layout: 96 requests in about 7 minutes. Streaming, temperature 0, thinking level LOW, up to 300 output tokens.
• Cached tokens: cachedContentTokenCount, as reported by Gemini for each request.
• Reusable prefix: the number of leading characters each request shares with the one before it, rendered as tools, then system instruction, then user message, and converted to tokens using Gemini's count. This depends only on the prompt, not on the provider. It's the most any prefix cache could reuse.
• Time to first token: from sending the request to the first streamed text or tool call.
• Cost: from Gemini's token counts and list prices: $0.75 per million uncached input tokens, $0.075 cached, $3.75 output (thinking counts as output), and $0.50 per million tokens per hour of explicit cache storage.

We also sent a 2-request smoke test before the run. When implicit caching produced no hits, we sent 12 follow-up requests. They varied one thing at a time: no streaming; no tools and no thinking setting; a prompt three times larger (13,100 tokens); and Google's newer Interactions API.

Results

The layout result is exactly what the prefix rule predicts. Moving the changing value to the end kept 99.4–99.7% of the prompt reusable. Putting it at the top cut that to 14%. The shuffled tools cut it to almost nothing, because the tools are the first thing in the prompt.

Table 5. All 96 requests, by layout (medians, 12 requests each). Every request and every answer is in results/requests.jsonl. 16 of the 96 responses were tool calls, all to lookup_order, for questions that named an order.

What it would have saved

The reusable prefix tells you what a working cache could save. If every request's reusable prefix had been read from cache at the cached price, here is the input cost of 1,000 requests. This is a projection from the measured prefixes, not a measurement:

Table 6. Layouts with a reusable prefix below 4,096 tokens save nothing. That isn't because 783 tokens wouldn't help, but because Gemini won't cache a prefix that short.

Why it happens

Practical advice

1. Design the prompt as layers: tools and instructions, then shared documents and examples, then per-user context, then conversation history, then the new message. Put anything with a clock, an ID or a counter in the last layer.
2. Make serialization deterministic. Sort tools by name and JSON keys alphabetically. Never build prompt text by iterating a set. Strip trailing whitespace consistently.
3. Pass dynamic context late. On Claude models that support it, a mid-conversation system message is designed for this: it adds instructions after the cached history instead of editing the top. Elsewhere, put it in the user message.
4. Choose explicit caching when the saving matters. On Gemini, create a cache for any large prefix you reuse more than about once an hour. On Claude, add cache_control at the end of the stable part.
5. Measure hits on every request. Log cached tokens next to total input tokens and alert when the ratio drops. Caching regressions don't fail; a deploy that adds a timestamp only shows up on the bill.
6. Warm up before fan-out. Send one request, wait for its first token, then send the parallel requests.
7. Check the minimum. A 3,000-token prompt caches on Claude Sonnet 5.5 (512 minimum) and never on Gemini 3.8 Flash (4,096) or Claude Haiku 4.5 (4,096).

What it looks like in code

Claude, with an explicit breakpoint at the end of the stable system prompt (Python SDK):

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    tools=sorted(TOOLS, key=lambda t: t["name"]),   # same order every time
    system=[{
        "type": "text",
        "text": INSTRUCTIONS + HANDBOOK,             # no dates, IDs or names in here
        "cache_control": {"type": "ephemeral"},      # cache everything up to this point
    }],
    messages=[{"role": "user", "content": f"Today's date: {today}\n\n{question}"}],
)
print(response.usage.cache_read_input_tokens, response.usage.cache_creation_input_tokens)

Gemini, with an explicit cache (REST, as used in our experiment):

# 1. Store the stable part once (default lifetime: 1 hour)
POST /v1beta/cachedContents
{ "model": "models/gemini-3.8-flash",
  "systemInstruction": { "parts": [{ "text": "<instructions + handbook + examples>" }] },
  "tools": [{ "functionDeclarations": [ ... ] }],
  "ttl": "600s" }

# 2. Each request sends only the new part and the cache name
POST /v1beta/models/gemini-3.8-flash:generateContent
{ "cachedContent": "cachedContents/abc123",
  "contents": [{ "role": "user", "parts": [{ "text": "Can I return a tent?" }] }] }
# → usageMetadata.cachedContentTokenCount: 5384

OpenAI caches automatically. Check usage.input_tokens_details.cached_tokens, and on GPT-5.6 and later also cache_write_tokens. The ordering rules are the same.

Limitations

• One provider, one day, one model. We measured Gemini 3.8 Flash on 7 October 2026. We didn't measure Claude or OpenAI; their sections come from documentation. Implicit caching may behave differently at other times, in other regions, or at higher traffic.
• Low volume. Requests went one at a time, about 4 seconds apart. Production traffic of many requests per second could get implicit hits that we didn't.
• Small samples. 12 requests per layout is enough to show 0 vs 12 hits, but not enough to estimate latency differences of a few hundred milliseconds.
• Reusable prefix is computed on our rendering order (tools, system, user). Gemini doesn't document its internal order. The before-and-after contrast holds either way, because the changed value was at the start or the end of a block.
• The playground estimates tokens at about four characters per token, and its cost model assumes steady traffic. Use it to compare layouts, not to forecast a bill.
• Prices change. Gemini 3.8 Flash prices are promotional and double on 1 January 2027.

Reproduce it

Everything runs with Node.js 20 or later and no dependencies. From production-deployment/prompt-caching/:

npm test                          # 17 tests for the analyzer and cost model
npm start                         # this article and the playground at http://127.0.0.1:4321
npm run experiment                # dry run: prints the plan and the worst-case cost
cp .env.example .env              # add GEMINI_API_KEY
npm run experiment -- --live      # 96 paid requests, about $0.40
npm run analyze                   # writes results/summary.json, requests.jsonl, report.md

The main run cost $0.38. With the smoke test and follow-ups, the total was about $0.46.

Sources

1. Anthropic, Prompt caching and Pricing, accessed 7 October 2026.
2. Google, Context caching (Interactions API), last updated 2 September 2026; Context caching (generateContent), 11 September 2026; Pricing, 6 October 2026.
3. OpenAI, Prompt caching and Pricing, accessed 7 October 2026.