Home / Semantic caching / Setting Similarity Thresholds for a Semantic Cache

Setting Similarity Thresholds for a Semantic Cache

A similarity threshold for a semantic cache is the cut-off that decides whether a new request is close enough to a stored request to reuse its answer. Set it too loose and the cache returns answers to questions nobody asked, such as a cancellation answer for a question about pausing. Set it too strict and almost nothing hits, so the cache saves no money. No universal value exists, because each embedding model spreads its vectors differently; Weaviate’s documentation says the right distance depends on the model and should be found by experimenting with your data. The reliable method is to collect real request pairs, label each as “same answer” or “different answer”, compute their distances, and sweep thresholds to see the trade-off between hit rate and wrong answers. Pick the loosest threshold whose wrong-answer rate you can accept, add scoping and verification for risky intents, and retune whenever the model or traffic changes.

What is a similarity threshold in a semantic cache?

A similarity threshold in a semantic cache is the minimum similarity, or maximum distance, a stored request must reach before its cached response is returned for a new request. It turns a ranked nearest-neighbour search into a yes-or-no decision.

A semantic cache embeds every incoming request and searches the stored requests for the nearest one. A plain vector search always returns something, even when the nearest match is unrelated. The threshold adds the missing judgment: if the nearest stored request is close enough, return its answer; otherwise call the model and store the new pair. The full flow is described in what is semantic caching.

Every request depends on that one comparison. A hit skips a model call, which saves cost and time. A wrong hit returns an answer to a different question, which can be worse than no cache at all. The threshold decides the balance between the two.

Because the decision is binary, small changes matter. A threshold that moves a little can flip many borderline requests from miss to hit. That sensitivity is why the value should come from measurement on your own traffic rather than a number copied from a tutorial. The first step is to understand what each kind of error costs.

Why does the threshold decide between wrong answers and wasted calls?

The threshold decides between wrong answers and wasted calls because loosening it lets more paraphrases hit, but also lets in near-misses that need a different answer. Tightening it does the reverse.

Two kinds of request pairs sit close together in embedding space. Paraphrases, such as “how do I reset my password” and “I forgot my password, what now”, should share an answer. Near-misses, such as “how do I cancel my plan” and “how do I pause my plan”, look almost the same but need different answers. Embeddings measure topical closeness, so near-misses can be as close as paraphrases.

A loose threshold accepts both. The hit rate rises, but some users get the wrong answer. A strict threshold rejects both. Wrong answers fall, but paraphrases miss too, and each miss costs a full model call. The relationship between closeness and meaning is covered in semantic similarity vs relatedness.

The two errors rarely cost the same. A wrong answer in a casual chatbot may be a mild annoyance. A wrong answer about refunds, dosage or account security can cause real harm. That asymmetry, not the hit rate alone, should set where the threshold sits. Before tuning, though, the threshold’s scale must be clear.

How do distance and similarity thresholds relate?

Distance and similarity thresholds relate as mirror images: a similarity threshold sets a minimum, while a distance threshold sets a maximum, and the conversion depends on the metric. Mixing them up silently turns a strict cache into a loose one.

Weaviate’s distance metrics reference defines cosine distance as 1 minus cosine similarity, with a range from 0 for identical vectors to 2 for opposite ones. A distance limit of 0.1 therefore accepts only matches with a cosine similarity of at least 0.9. In Weaviate, a vector query takes a distance argument as the maximum accepted distance, and results beyond it are dropped.

Other metrics have no fixed range. Weaviate’s reference notes that the dot metric returns the negative dot product and that squared Euclidean distance runs from 0 upward. Thresholds on those scales depend on vector lengths, so they are harder to reason about. With normalized embeddings, cosine distance is the clearest choice. The metrics are compared in cosine vs dot product vs Euclidean distance, and the conversions can be checked in the vector similarity calculator.

Caching libraries also differ in direction. Some take a similarity that must be exceeded; others take a distance that must not be exceeded. Read each tool’s definition before copying a value, as the differences in GPTCache show. With the scale fixed, the threshold can be tuned on data.

How do you build labeled data for tuning a cache threshold?

You build labeled data by pairing real requests with their nearest stored requests and marking each pair as “same answer” or “different answer”. The labels define what a correct hit means for your application.

Start from logs. Take a sample of incoming requests, find the nearest earlier request for each with the same embedding model the cache will use, and record the distance. That gives realistic pairs at realistic distances, including the borderline ones that matter most. Random pairs are too far apart to teach anything.

Label each pair by whether one cached answer would serve both requests, not by whether the questions sound alike. “What is your refund window” and “how long do I have to return something” share an answer. “Can I return opened items” and “can I return unopened items” may not. Have someone who knows the product make these calls, and keep a written rule for borderline cases.

Make sure the set includes enough near-misses. They are the pairs that cause wrong answers, and a set made only of paraphrases will push the threshold too loose. Group pairs by intent, such as billing, account and product questions, so per-intent behaviour can be checked later. Building judged sets is covered in search test sets and relevance judgments. With labels in hand, the threshold can be swept.

How do you sweep thresholds to choose a value?

You sweep thresholds by applying each candidate value to the labeled pairs and counting correct hits, wrong hits and missed paraphrases at each one. The result is a table of trade-offs to choose from.

In the illustrative set, six pairs should hit, at cosine distances 0.04, 0.07, 0.09, 0.12, 0.16 and 0.22. Four should not, at 0.11, 0.18, 0.25 and 0.31. At a limit of 0.10, three correct hits pass and nothing wrong does: precision 1.0, recall 3 of 6, or 0.5. At 0.15, four correct and one wrong pass: precision 0.8, recall 0.67. At 0.20, five correct and two wrong pass: precision 0.71, recall 0.83.

def sweep(pairs, thresholds):
    """pairs: list of (cosine_distance, should_hit). One row per threshold."""
    total_should = sum(1 for _, ok in pairs if ok)
    rows = []
    for t in thresholds:
        hits = [ok for d, ok in pairs if d <= t]
        correct = sum(hits)
        rows.append((t, correct, len(hits) - correct,
                     correct / len(hits) if hits else 1.0,
                     correct / total_should))
    return rows

Precision here is the share of hits that are correct, so one minus precision is the wrong-answer rate among hits. Recall is the share of hit-worthy pairs that actually hit. The function returns both for every candidate. The metrics behave like their search counterparts in precision at k and recall at k. The table shows options; choosing one needs a rule.

How do you pick the operating point for a cache threshold?

You pick the operating point by setting the highest wrong-answer rate you can accept, then choosing the loosest threshold that stays within it. The loosest acceptable value gives the most savings for that risk.

In the illustrative sweep, a team that accepts no wrong hits must use 0.10 and gets half the possible hits. A team that accepts one wrong hit in five could use 0.15 and get two thirds. No threshold gives both full precision and full recall, because the near-miss at 0.11 sits between paraphrases at 0.09 and 0.12.

The acceptable rate comes from the cost of errors. Multiply the wrong-hit rate by the harm of a wrong answer, and compare it with the saving per hit. Weaviate’s documentation gives no universal value for this reason: it says the distance depends on many factors, including the vectorization model, and should be found by experimenting with your data.

Start strict and loosen in small steps while watching wrong answers in production. A strict cache that misses costs money; a loose cache that answers wrongly costs trust. The money is easier to recover. The economics of caching are discussed in semantic caching for LLM apps. Once chosen, the threshold goes into the lookup query.

How do you apply a cache threshold in Weaviate?

You apply a cache threshold in Weaviate by passing a distance limit to the vector query against the cache collection, along with filters that scope the lookup. If no object comes back, it is a miss.

from weaviate.classes.query import Filter, MetadataQuery

cache = client.collections.use("LLMCache")

def lookup(query_vector, tenant_scope, prompt_version, max_distance=0.10):
    res = cache.query.near_vector(
        near_vector=query_vector,
        distance=max_distance,          # cosine distance; 0.10 means similarity of at least 0.90
        limit=1,
        filters=Filter.by_property("scope").equal(tenant_scope)
                & Filter.by_property("prompt_version").equal(prompt_version),
        return_metadata=MetadataQuery(distance=True),
    )
    return res.objects[0] if res.objects else None

The distance argument drops any match beyond the limit, so the code only needs to check whether an object came back. Returning the distance in the metadata lets the application log how close each hit was, which feeds later tuning. The 0.10 default is a placeholder for the value your sweep produces.

The filters matter as much as the threshold. A cached answer for one customer, language or prompt version should never serve another, however close the vectors are. Scoping the lookup removes whole classes of wrong hits before distance is considered. Filtering is covered in metadata filtering. Even with scoping, some intents need more than a single cut-off.

When is a single similarity threshold not enough?

A single similarity threshold is not enough when paraphrases and near-misses overlap in distance, which is common for short questions that differ by one key word. Then extra guards are needed.

Per-intent thresholds help when some topics are riskier than others. Classify the request first, then use a strict threshold for billing or account changes and a looser one for general questions. Some intents should never be cached, such as anything that depends on the user’s current account state.

Verification adds a second check on candidate hits. A cross-encoder reads the new request and the cached request together and scores whether they mean the same, which handles negation and single-word differences better than vector distance, as explained in cross-encoder rerankers. A small language model can do the same at higher cost. Verification only runs on candidate hits, so it adds little to misses.

Cached answers also go stale. A correct hit can still return an outdated answer if the underlying facts changed. Expiry and invalidation are covered in semantic cache invalidation. Gateways that provide caching as a service expose the same settings, as described in AI gateway semantic caching. A threshold that worked at launch also needs checking over time.

How do you monitor and retune a cache threshold over time?

You monitor a cache threshold by logging the distance of every hit, sampling hits for human review, and rerunning the sweep when traffic, content or the embedding model changes. A threshold is only valid for the data it was tuned on.

Log each lookup with its nearest distance, whether it hit and the intent. Review a regular sample of hits, especially those close to the limit, and record which were wrong. A rising share of wrong hits near the limit is the signal to tighten. A large share of misses just beyond the limit that reviewers judge correct is the signal to loosen.

An embedding model change resets everything. A new model spreads vectors differently, so the old threshold has no meaning, and stored request vectors must be recomputed too, as explained in re-embedding when the model changes. Rerun the labeled sweep before switching.

Prompt and product changes matter too. A new prompt version can change answers for the same request, which is why the lookup should filter on prompt version. Semantic caching also differs from provider-side prompt caching, which reuses computation rather than answers and needs no threshold, as compared in semantic caching vs prompt caching.

A similarity threshold decides whether a semantic cache reuses an answer, and it trades hit rate against wrong answers. Fix the scale first, cosine distance being 1 minus similarity in Weaviate, then label real request pairs as same-answer or different-answer and sweep candidate values. Choose the loosest threshold whose wrong-answer rate you can accept, starting strict. Scope every lookup by tenant and prompt version, use stricter rules or verification for risky intents, and rerun the sweep whenever the model, the prompts or the traffic change.