Home / Semantic caching / What Is Semantic Caching? How It Cuts LLM Cost and Latency

What Is Semantic Caching? How It Cuts LLM Cost and Latency

Semantic caching is a response-reuse method that returns a stored answer when a new prompt means the same thing as an earlier prompt. It embeds each prompt, searches stored prompt vectors, and compares the closest match with a threshold before calling a large language model (LLM). A hit skips generation, which can reduce latency and model usage. A miss calls the model and stores the new prompt-response pair. Safe use depends on scoped cache entries, explicit invalidation, and measured wrong-hit rates. Semantic caching therefore suits repeated, stable questions better than personalized, time-sensitive, or conversation-dependent requests.

What is semantic caching in an LLM application?

Semantic caching in an LLM application stores past prompts as embeddings with their responses, then reuses a response for a new prompt that is close enough in meaning. A stored record usually contains prompt text, a prompt vector, the response, and metadata such as user scope or creation time. RedisVL documents this pattern through its SemanticCache interface. The interface checks an embedded query against stored entries, returns matching responses, and stores new prompt-response pairs after misses.

The word semantic describes a meaning-based match rather than a text match. “What is the capital of France?” and “What actually is the capital of France?” use different strings but can retrieve the same stored answer. RedisVL uses this pair in its current guide. The lookup first turns text into an embedding, then performs semantic search over past prompts instead of documents. Exact punctuation and capitalization do not define the result.

A semantic cache is not retrieval-augmented generation (RAG). RAG retrieves evidence and still asks a model to generate an answer. A semantic-cache hit returns an earlier answer before that generation step. A cache miss may continue into RAG, after which the generated answer becomes a new cache entry. The guide to semantic search in RAG covers retrieval quality and grounding. The separate mechanisms can occupy the same request path without becoming the same operation.

Meaning-based reuse increases reach beyond an exact cache, but it also creates uncertainty. GPTCache documents false positives, where a hit should have been a miss, and false negatives, where a miss should have been a hit. That uncertainty explains why model-call savings alone do not establish that a semantic cache is useful.

Why do LLM applications use semantic caching?

LLM applications use semantic caching to avoid repeated generation for requests that ask the same stable question in different words. GPTCache explains that exact matching has a low hit rate when LLM queries vary. Embedding-based matching captures paraphrases that produce different hashes. Support questions, repeated classifications, and fixed-policy explanations often contain this kind of semantic repetition. Detailed choices about semantic caching for LLM apps include what to cache and where the cache sits.

A hit replaces a model call with an embedding and lookup. Fu Bang presented “GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings” at NLP-OSS 2023. Bang reported that GPTCache with OpenAI service could answer two to ten times faster on a hit. The result is an author-reported system measurement, not a general promise. Misses still pay for lookup plus generation, so traffic shape determines the net result.

RedisVL’s “Cache LLM Responses” guide records one example run at about 1.35 seconds for an uncached call and about 0.042 seconds on average with caching across ten calls. Those values describe that guide’s setup. They do not predict another model, network, embedding service, or cache store. Still, the measurement shows the mechanism: a cache hit removes generation time rather than making generation itself faster.

Hit opportunities can be substantial in repetitive datasets. Sajal Regmi and Chetan Phakami Pun tested 2,000 queries against 8,000 cached pairs. Their 2024 paper “GPT Semantic Cache” reports four categories with hit rates from 61.6% to 68.8% at a cosine-similarity threshold of 0.8. Those hits reduced model calls, but some were wrong. Cost and latency gains therefore lead directly to the lookup process and its correctness controls.

How does semantic caching work from prompt to response?

Semantic caching works by embedding a prompt, searching stored prompt vectors within its permitted scope, testing the nearest match, and either returning a stored response or calling the model. The same embedding model must represent stored and incoming prompts so their vectors share one space. Weaviate’s vector-search documentation states that distance values depend on factors including the vectorization model. Changing that model changes the meaning of the comparison.

A production lookup has six ordered operations. Scope comes before nearest-neighbor search because the closest prompt from another user may contain an answer the caller must not receive. Weaviate documents filters combined with vector search, while RedisVL documents filterable fields such as user_id. An approximate nearest neighbor index can accelerate the search, but indexing method does not replace access control.

1. Normalize the cache request into the prompt text and metadata that determine reuse.
2. Resolve the request scope from trusted user, tenant, route, and version fields.
3. Embed the incoming prompt with the model used for stored prompt vectors.
4. Search stored prompts for the nearest permitted vector, usually with a limit of one.
5. Compare the nearest score with the configured similarity or distance threshold.
6. Return or write the response by serving a hit, or calling the LLM and storing a miss.

Weaviate exposes the core lookup through near_text, a result limit, a maximum distance, and optional filters. A semantic cache adds response storage and write-back around that vector search. RedisVL packages the operations as check() and store(). A gateway can apply the same sequence before requests reach a model provider.

Every request pays for steps before the threshold decision. A miss therefore adds embedding and lookup work to the original model path. A hit avoids the model call but returns an old response whose scope and freshness must still be valid. Threshold choice separates those two paths.

How does a semantic cache decide that two prompts match?

A semantic cache decides that two prompts match by comparing their nearest similarity score or distance with a configured threshold. A threshold has meaning only with its metric and direction. Weaviate documents cosine distance as one minus cosine similarity, with zero for identical vectors and a range from zero to two. Smaller distance is closer, while larger cosine similarity is closer. A bare value such as 0.9 is ambiguous without naming the scale.

RedisVL uses cosine distance for distance_threshold. Its guide begins at 0.1, equivalent to cosine similarity 0.9 under that definition. Widening the distance threshold to 0.5 lets an indirect question about the country containing Nice match a cached answer about France’s capital. This example demonstrates both extra recall and extra risk. The cache has accepted a broader relationship, not proven that two prompts need one answer.

Microsoft Azure API Management uses another score. Its score-threshold runs from 0.0 to 1.0, lower values require higher semantic similarity, and its documentation suggests starting at 0.05. The same documentation warns that values above 0.2 may create cache mismatches. Microsoft does not define that score as cosine distance, so it cannot be converted by subtracting it from one.

One threshold cannot guarantee correctness. Luis Gaspar Schroeder and nine coauthors reported in the 2025 paper “vCache: Verified Semantic Prompt Caching” that correct and incorrect hits have highly overlapping similarity distributions. Unless a threshold is 1.0, static matching retains some risk of an incorrect response. Weaviate likewise advises experimenting because suitable distance depends on the vectorization model and data. The guide to similarity thresholds for caching covers labelled pairs, sweeps, and route-specific limits.

Threshold tuning therefore needs prompts that should share an answer and prompts that must not. Entity changes, negation, user identity, and time references deserve deliberate negative pairs. The comparison between cosine similarity and cosine distance matters before any numeric threshold is copied into configuration.

How is semantic caching different from prompt caching and exact caching?

Semantic caching returns a stored response for a meaning-equivalent prompt, while exact caching requires an identical request and prompt caching reuses a processed prefix before generating a fresh response. These three caches reuse different work. They can run in one system because a prompt-cache hit still executes the model, while a response-cache hit does not.

Cloudflare AI Gateway documents exact response caching as of September 30, 2026. Its key is a SHA-256 hash that includes the provider, endpoint, model, authorization header, and full request body. Cloudflare says caching currently applies only to identical requests and describes semantic search for caching as planned. That documented boundary corrects vendor lists that classify its current cache as semantic.

OpenAI’s prompt-caching documentation describes reuse when requests share a prompt prefix. For GPT-5.6 and later, the minimum cacheable prefix is 1,024 tokens. The model still produces a response after cached prefix processing. Semantic caching instead returns the full stored response and avoids that generation call. The complete guide to semantic caching vs prompt caching covers provider behavior, billing, and implementation details.

The three mechanisms differ most in wrong-answer risk. Exact response caching reuses only an identical key, although stale content remains possible. Prompt caching reuses identical prefix computation and generates a new completion. Semantic caching deliberately accepts non-identical inputs, so a false positive may return an answer to a nearby but different question. The layers should be selected by reuse target rather than treated as interchangeable names.

A common order checks the exact-response cache first, then the semantic cache, and calls the model only after both miss. Prompt caching may still reduce work on that final model call. This layered order preserves cheap exact hits before paying for an embedding lookup.

Which tools and gateways support semantic caching?

Tools supporting semantic caching include vector-database building blocks, application libraries, and gateway policies. Weaviate supplies filtered vector lookup and maximum-distance controls rather than a dedicated response-cache object. An application stores prompt vectors, responses, and scope metadata in a collection. Libraries such as GPTCache and RedisVL package more of that lifecycle. Gateways such as Kong and Azure API Management place lookup before shared model routes.

GPTCache is an MIT-licensed library with embedding generation, cache storage, vector-store adapters, cache management, and similarity evaluation. Its README lists Weaviate among supported vector stores. RedisVL provides a SemanticCache abstraction over Redis with threshold, TTL, filters, check, and store operations. The dedicated GPTCache guide covers its components and implementation path.

Kong’s AI Semantic Cache plugin documents two backend families for vector storage. Azure API Management exposes semantic-cache lookup and store policies with score threshold, duration, message-count, and vary-by controls. These gateway features centralize mechanics for multiple model routes, while application code still decides which requests are safe to reuse. The AI gateway semantic caching guide covers that division of responsibility.

Cloudflare belongs in an exact-cache row, not a current semantic-cache row. Its docs state that semantic search for caching is planned. Product categories therefore need checks against official documentation rather than copied vendor lists. Placement also changes observability: an application cache sees domain context, while a gateway sees traffic across services.

Tool support makes matching possible, but it does not make every stored answer safe. Scoping, freshness, and conversation state remain application-level facts even when a gateway enforces their fields.

When does semantic caching return a wrong or stale answer?

Semantic caching returns a wrong or stale answer when vector closeness hides a factual difference, scope omits the caller, source information changes, or conversation context is missing. Azure API Management warns that semantic caching may surface incorrect, outdated, or unsafe responses. A threshold is only one defense because embeddings group related language, not guaranteed answer equivalence.

Near-but-different prompts form the basic false hit. A country, product code, date, or negation may change the required answer while leaving most words unchanged. RedisVL’s widened-threshold example links an indirect question about Nice with the capital of France. That result is suitable only because the geographic relation preserves the answer. A similar relation with a changed entity may not.

Scope errors are more serious than ordinary relevance errors. A shared entry for “Where is my order?” may return another person’s status when user identity is absent from lookup filters. Weaviate combines metadata filters with vector search, RedisVL documents user_id filter fields, and Azure documents vary-by. Trusted tenant, user, route, model, and prompt-version fields should partition entries before similarity search.

Freshness needs explicit policy. RedisVL gives entries no TTL by default, and its check() refreshes the TTL of every matched entry. That sliding window means a popular stale answer may never expire through TTL alone. Prompt changes, policy changes, and source-data updates need explicit semantic cache invalidation, such as versioned namespaces or targeted purges.

Multi-turn requests need more than the last message. Regmi and Pun identify lack of multi-turn handling as a limitation, while Azure provides max-message-count to stop caching beyond a configured conversation length. A reply such as “What about tomorrow?” has little stable meaning without prior turns. Either include validated conversation state in identity or bypass semantic caching for that route.

Cache writes create another boundary. Azure recommends considering its content-safety policy with prompt shield. A poisoned or unsafe generated answer should not become a reusable entry. Embedding-model changes also require rebuilding or separating entries because vectors produced by different models do not share a dependable score scale.

How do you measure a semantic cache’s hit rate and error rate?

Measure a semantic cache’s hit rate as all hits divided by requests, and measure error rate as wrong hits divided by requests. Schroeder and coauthors define hit rate as true positives plus false positives over all prompts, and error rate as false positives over all prompts. A separate wrong-share-of-hits value shows how many served cache responses were wrong. Reporting all three prevents a high hit rate from hiding unsafe reuse.

A labelled evaluation sample is required because production logs rarely state whether two prompts deserved the same answer. Reviewers or a validated evaluator mark each hit as acceptable or wrong. Thresholds are then swept over the same pairs. GPTCache names hit ratio, latency, and recall as core metrics. The wider guide to evaluating semantic search explains judgments and segmented analysis.

Regmi and Pun’s Table 1 provides a worked example. Across four categories, their cache returned 1,322 hits from 2,000 queries, a 66.1% hit rate. GPT-4o mini judged 57 of those hits wrong. That equals 4.3% of hits and 2.85% of all queries. Per-category valid shares ranged from 92.5% for Python basics to 97.3% for network technical support.

Those figures belong to one dataset, embedding model, cache, and threshold. They do not establish a production target. The variation between categories is the useful finding: one global average can hide a weaker route. Segment metrics by route, tenant, question type, cache age, and threshold. Latency should include lookup time on misses as well as saved generation time on hits.

When is semantic caching worth adding to an LLM application?

Semantic caching is worth adding when requests repeat in meaning, answers remain valid across callers and time, and measured hit rate exceeds the lookup cost without an unacceptable wrong-hit rate. FAQ answers, stable support explanations, and repeated classification prompts often meet those conditions. Unique creative prompts, personalized account answers, current prices, and context-dependent agent turns often do not.

Latency has a simple break-even rule. Let L be model-call latency, c be cache-path latency, and h be hit rate. Expected latency is c + (1 - h) × L. A cache improves average latency when h > c / L. This formula assumes every eligible request pays the lookup cost and every hit avoids the model call. The detailed search latency budget helps separate embedding, retrieval, network, and model time.

The calculator starts with two source sets for illustration. RedisVL’s guide supplies 1,347 milliseconds for an uncached call and 42 milliseconds for its cached path. Regmi and Pun supply a 66.1% hit rate and a 4.31% wrong share among hits. Combining different systems does not create a benchmark. It demonstrates the formula while keeping every input visible and editable.

Correctness stays beside speed in the decision. Expected wrong answers per 1,000 requests equal 1,000 × h × w, where w is wrong hits divided by all hits. Lowering a threshold may increase hits and wrong answers together. A profitable cache that leaks user data or returns stale policies is still a failed system.

Semantic caching turns repeated meaning into response reuse, but useful reuse requires more than a nearby vector. Scope protects callers, invalidation protects freshness, and labelled evaluation protects answer quality. Add the cache when measured repetition repays its lookup cost and wrong-hit risk fits the route. Otherwise, exact response caching or prompt caching keeps a narrower and safer reuse boundary.