Home / Semantic caching / Semantic Caching vs Prompt Caching vs Exact Caching

Semantic Caching vs Prompt Caching vs Exact Caching

Exact caching, semantic caching and prompt caching all cut the cost of LLM applications, but they reuse different things. An exact cache stores a full response under a hash of the exact request and returns it only for an identical request. A semantic cache stores responses under an embedding and returns one when a new request means nearly the same thing, which raises hit rates but can return a wrong answer. A prompt cache, which some providers call context caching, stores the model’s computed state for a repeated prompt prefix. The model still generates a fresh answer, so correctness is unaffected, and only the prefix is billed at a discount. With Anthropic’s documented 5-minute rates, a 10,000-token prefix reused 100 times costs 111,500 token-equivalents instead of 1,000,000. Most production systems combine them: exact first, then semantic, with prompt caching inside every model call.

How do semantic, prompt and exact caching differ?

They differ in the cache key and the stored value: exact caching keys on the exact request text, semantic caching on its meaning, and prompt caching on an identical prompt prefix. The first two store answers; the third stores computation.

An exact cache and a semantic cache sit in front of the model. On a hit, the application returns a stored response and never calls the model. A prompt cache sits inside the model provider’s serving system. Every request still reaches the model, but a long shared prefix, such as instructions, tool definitions or a document, is not processed again.

That difference sets the risk. A stored answer can be wrong for a new request that only looks similar. A cached prefix cannot, because the model still reads the new part of the prompt and writes a new answer. The basics of the first two are covered in what is semantic caching and cache hits and misses.

How does exact caching work?

Exact caching hashes the full request, including the prompt, model name and parameters, and looks up a stored response under that hash. It is simple, fast and never returns an answer meant for a different request.

A key-value store such as Redis holds the hash and the response, with an expiry time. Any difference in the request, even an extra space, produces a different hash and a miss. Normalising input first, such as trimming whitespace and lower-casing where safe, raises the hit rate a little. Exact caching suits repeated automated calls, fixed prompts and popular identical questions.

Where the cache lives matters. A cache inside a single node or pod, in front of a database or a model, is the simplest option, but each replica keeps its own copy. Hit rates fall as replicas multiply, and copies disagree after updates. A shared cache serving all replicas avoids both problems, at the cost of a network hop.

How does semantic caching work?

Semantic caching embeds each incoming request, searches stored requests for the nearest one, and returns its response if the similarity passes a threshold. It catches paraphrases that an exact cache misses.

from weaviate.classes.query import MetadataQuery

cache = client.collections.use("AnswerCache")
hit = cache.query.near_vector(
    near_vector=query_vector,
    distance=0.05,  # only accept very close matches
    limit=1,
    return_metadata=MetadataQuery(distance=True),
)
answer = hit.objects[0].properties["answer"] if hit.objects else None

A vector database such as Weaviate stores the request embeddings and responses, and a distance limit on the query acts as the threshold. “How do I reset my password” can then reuse the answer to “I forgot my password”. Full designs, including where the cache sits in the request path, are in semantic caching for LLM apps.

The risk is false hits. Two requests can be close in embedding space yet need different answers, such as “enable two-factor login” and “disable two-factor login”. The threshold sets the trade-off between hit rate and wrong answers, as discussed in similarity thresholds for caching. Stored answers also go stale, which is covered in semantic cache invalidation.

How does prompt caching work?

Prompt caching stores the model’s intermediate attention state for a prompt prefix, so a later request that starts with the same prefix skips recomputing it. The provider handles the cache; the application only structures prompts to share prefixes.

Transformer models compute key and value tensors for every input token. For a long system prompt or document that many requests share, that work repeats. Prompt caching keeps those tensors for a while and reuses them when the same prefix arrives again. Requests must match the cached prefix exactly from the first token, so stable content goes first and variable content last.

Providers implement it differently. Anthropic uses explicit cache breakpoints in the request, with a default lifetime of 5 minutes and an optional 1-hour lifetime at a higher write price. OpenAI applies caching automatically to long prompts. Google’s Gemini API calls the feature context caching and offers explicit caches with a set lifetime. Check each provider’s current documentation for minimum lengths and prices.

How much does each method save?

Exact and semantic caches save the whole model call on a hit, while prompt caching saves part of the input cost and latency on every call that reuses a prefix. The real savings depend on traffic patterns, not on the method alone.

For prompt caching, the arithmetic is direct. Using Anthropic’s documented rates, the first request writes the prefix at 1.25 times the base input price and later requests read it at 0.1 times. A 10,000-token prefix used 100 times within the cache lifetime costs 12,500 plus 99 times 1,000, or 111,500 token-equivalents. Without caching it costs 1,000,000. Output tokens and the new part of each prompt are billed as usual.

For response caches, savings equal the hit rate times the cost of a full call. Exact hit rates are usually low for free-text user input, and semantic caching raises them. Every semantic hit also needs checking: a wrong cached answer can cost more in support or trust than the model call it saved.

Which method is most likely to return a wrong answer?

Semantic caching carries the most correctness risk, because it deliberately returns an answer written for a different request. Exact caching risks only staleness, and prompt caching risks neither, because the answer is still generated.

Semantic cache errors come from near-miss requests: opposites, different products, different dates or different users. Answers that depend on personal context, such as account details, should never be shared across users through a semantic cache. Scope the cache by user, tenant or product where answers differ, as described in multi-tenant semantic search.

Both response caches can return stale answers after the source data changes. Set expiry times that match how often answers change, and invalidate entries when the underlying documents update. Prompt caching has no staleness issue in the same sense, as changing the prefix simply creates a new cache entry.

When should you use a shorter context instead of prompt caching?

Use a shorter context instead of prompt caching when the long prefix changes often or requests arrive too rarely to reuse the cache. The same holds when only a small part of the context is relevant. Caching a prompt that is too long still costs latency and some money.

A cache that expires before the next request never pays off. Low-traffic applications with a 5-minute lifetime pay the write premium without enough reads. Prefixes that change per user or per hour also miss constantly. In those cases, trimming the prompt saves more than caching it.

Long context can also hurt answer quality when most of it is irrelevant. Retrieving only the passages a question needs, with semantic search, keeps prompts short and focused, as described in semantic search in RAG. Caching then applies to the stable instructions, while retrieval supplies the variable part.

How do you combine the three caching methods?

Combine them in layers: check an exact cache first, then a semantic cache with a strict threshold, and on a miss call the model with a prompt structured for prompt caching. Each layer handles what the previous one cannot.

The exact layer is cheapest and safe, so it goes first. The semantic layer catches paraphrases, scoped and thresholded to keep wrong answers rare. On a full miss, the model call places stable instructions, tools and shared documents at the start of the prompt so the provider can reuse them. Store the new response in both caches after any checks.

Gateways can manage these layers centrally across applications, as covered in AI gateway semantic caching. Libraries such as GPTCache implement exact and semantic layers in application code, as described in GPTCache. Measure each layer’s hit rate, wrong-answer rate and cost separately.

Exact caching reuses a full response for identical requests and is always safe but rarely hits. Semantic caching reuses a response for requests that mean nearly the same thing, raising hit rates at the risk of wrong answers that thresholds and scoping must control. Prompt caching, also called context caching, reuses the computed prompt prefix while the model still answers. With Anthropic’s documented rates, it cuts the cost of a 10,000-token prefix used 100 times to 111,500 token-equivalents from 1,000,000. Layer exact, semantic and prompt caching, and shorten context when prefixes change often or traffic is too sparse to reuse them.