Prompt Caching: Stop Paying to Re-read the Same Context

A language model has no memory between calls, so every request re-reads the whole prompt from the first token. If your app sends the same large system prompt, the same retrieved document, or a long and growing chat history on every call — and most apps do — you are paying to process those identical tokens over and over again.

Prompt caching ends that waste. It lets the model do the expensive work of reading a stable prefix once, store the result, and reuse it on later calls that start with the same text. For workloads built on a big fixed context — RAG, agents, document chat — it is one of the largest cost and latency wins available, and it is mostly a matter of flipping it on correctly.

Why you keep paying for the same tokens

Generating a response has two phases: reading the prompt (prefill) and writing the answer (decode). Prefill is pure overhead that scales with prompt length, and without caching you repeat it in full on every request — even when 95% of the prompt has not changed since the last call.

• Long system prompts and tool definitions. An agent ships the same multi-thousand-token instructions and tool schemas on every single turn.
• RAG context. The same handful of retrieved documents are re-read for every question asked against them.
• Growing chat history. Each new turn re-sends and re-processes the entire conversation so far, which gets quadratically expensive.
• Batch over one document. Asking twenty questions about the same long contract means reading that contract twenty times.

Cache the prefix, not the whole thing

Caching works on prefixes. The model can reuse cached work only up to the first token that differs from a previous call, so the rule is simple: put everything stable at the front and everything that changes at the very end. System prompt and documents first; the user's new question last.

Hosted providers expose this directly — you mark where the cacheable prefix ends, and a cache hit on later calls costs a small fraction of the normal rate and returns faster. Self-hosted engines like vLLM do it automatically with prefix caching: identical leading tokens reuse the stored KV state instead of recomputing it.

Turning it on

With a hosted API you add a marker at the boundary of the static content. Everything up to it is cached; everything after is fresh each call:

# Cache the big static prefix so you stop re-paying to process it every call.
import anthropic
client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-sonnet-4-6",
    system=[
        {"type": "text", "text": SYSTEM_PROMPT},
        {"type": "text", "text": BIG_DOCUMENT,
         "cache_control": {"type": "ephemeral"}},   # cache everything up to this point
    ],
    messages=[{"role": "user", "content": question}],  # only this changes per call
)
# the first call writes the cache; later calls with the same prefix reuse it
# for a fraction of the cost and latency

The first call costs a little extra to write the cache; every later call that shares the prefix reads it cheaply. The break-even is usually two or three reuses, which most real workloads clear instantly.

Without caching, a long prompt is a tax you pay on every single request. With it, you pay once and collect the savings for as long as the context stays put.

Making caching actually hit

• Static first, dynamic last. Order the prompt so the unchanging parts lead; a single changed token near the front busts the entire cache behind it.
• Keep the prefix byte-identical. Caches match on exact prefixes, so a timestamp or a reordered tool list quietly turns every call into a cache miss.
• Mind the TTL. Provider caches expire after minutes of inactivity; high-traffic prefixes stay warm, rarely-used ones pay full price again.
• Cache the big, reused things. System prompts, tool schemas, long documents, and few-shot examples are ideal; tiny or one-off prefixes are not worth it.
• Self-hosting? Turn on automatic prefix caching in your serving engine and structure prompts the same way — it costs you nothing and helps every shared prefix.

The bottom line

Prompt caching is close to free money for any app with a large, stable context. Move the unchanging content to the front, mark or enable caching, keep the prefix identical across calls, and watch the cost and latency of repeated context collapse — often by most of the prompt's price.

It pairs naturally with everything else you do to cut cost: route easy queries to small models, cache their stable prompts, and you are no longer paying full freight for either the model or the tokens. The cheapest token is the one you only read once.