Home / Semantic caching / GPTCache Explained: How It Works and Alternatives

GPTCache Explained: How It Works and Alternatives

GPTCache is an open-source Python library from Zilliz that adds a semantic cache in front of large language model calls, returning a stored answer when a new prompt is similar enough to an earlier one. It is built from swappable modules: an LLM adapter, an embedding generator, cache storage, a vector store, a cache manager for eviction, and a similarity evaluator. That evaluator decides what counts as a hit. GPTCache supports several vector stores, Weaviate among them, and is MIT licensed. Its last PyPI release, 0.1.44, dates from August 2024, though the GitHub repository saw new commits in September 2026. In this site’s test, neither cosine distance nor an off-the-shelf duplicate-question model separated true paraphrases from look-alike questions. Tune the evaluator before trusting hits.

What is the GPTCache library?

The GPTCache library is a tool for creating a semantic cache for LLM queries, storing model responses and serving them again when a similar request arrives. Zilliz maintains it on GitHub under the MIT license; Zilliz also develops the Milvus vector database.

The project describes its goal as cutting LLM API costs and speeding up responses. Its documentation claims cost savings of up to 10 times and speed gains of up to 100 times; those are best-case figures that depend entirely on how often requests repeat. A paper by Fu Bang, presented at the NLP-OSS 2023 workshop, describes the system as an open-source semantic cache that stores LLM responses to address slow and expensive API calls.

Semantic caching itself is the broader idea; GPTCache is one implementation. What is semantic caching explains the concept, and semantic caching for LLM apps covers design choices that apply to any implementation.

Why should you use GPTCache?

You should use GPTCache to avoid paying for, and waiting on, repeated LLM calls when users ask the same or similar questions, without building a cache layer yourself. It packages the parts into one library.

The README lists the main benefits: lower expense from fewer API calls, faster responses for cached queries, a stand-in for the LLM during development and testing, and better availability when the provider rate-limits or fails. The Thoughtworks Technology Radar placed GPTCache in its “Assess” ring in September 2023. It cited the same two reasons: better performance from fewer external calls, and lower operating cost from caching similar responses.

The value depends on traffic. A support assistant answering the same policy questions all day benefits; a coding assistant where every prompt is unique benefits little. Measure how often your prompts repeat before adding any cache.

How does GPTCache work?

GPTCache works by embedding each incoming prompt, searching a vector store for similar cached prompts, and asking a similarity evaluator whether the best match is close enough. On a hit it returns the cached answer; on a miss it calls the LLM and stores the new one. Each step is a module you can replace.

• LLM adapter: wraps the model API, such as OpenAI or LangChain, so caching needs few code changes.
• Embedding generator: turns prompts into vectors, with options including ONNX, OpenAI, Cohere, Hugging Face, and SentenceTransformers; disabling it gives a keyword-matching cache.
• Cache storage: keeps questions and answers in SQLite, PostgreSQL, MySQL, or DuckDB.
• Vector store: finds the most similar cached prompts, using Weaviate, Milvus, Zilliz Cloud, FAISS, Hnswlib, pgvector, Chroma, Qdrant, and others.
• Cache manager: handles eviction with LRU, FIFO, LFU, or random policies, in memory or through Redis or memcached for distributed setups.
• Similarity evaluator: decides the hit, using vector distance, a duplicate-detection model, exact match, or BM25.

The README is candid about one limit: in-memory eviction counts entries rather than bytes, which can lead to out-of-memory errors. It also offers a temperature setting: at 0 the cache is always searched, and at 2 it is always skipped.

Why does the similarity evaluator matter most?

The similarity evaluator matters most because it alone decides whether a near match is the same question, and a wrong decision returns a confident, wrong answer. GPTCache’s README names the two errors as false positives on hits and false negatives on misses.

GPTCache offers a distance-based evaluator and a model-based one built on its ALBERT duplicate-detection model. We compared the two ideas on this site’s cache test pairs: 180 paraphrase pairs that should share an answer and 30 near-miss pairs that should not, such as “enable” versus “disable” two-factor authentication. For distance we used all-MiniLM-L6-v2 cosine similarity. For the model, we used a public Quora duplicate-question classifier as a stand-in, because GPTCache’s ALBERT model needs an ONNX runtime.

Neither evaluator separated the groups. A threshold strict enough to reject 90 percent of near-misses kept only 5 percent of paraphrases with cosine distance and 9 percent with the duplicate classifier. Both rank AUC values were under 0.5, meaning near-misses tended to score higher than true paraphrases. The classifier rated “reset my password” against “reset my PIN” at 0.953.

The classifier was also very strict: most of our loose paraphrases, like “Can’t log in, need a new password” against “Reset password steps”, scored near zero. That is safe but leaves few hits. The lesson matches Redis researchers’ 2025 finding that a cache-specific model fine-tuned on in-domain pairs beat larger general models. Similarity thresholds for caching covers threshold tuning.

How do you set up GPTCache?

You set up GPTCache by installing the package, initializing a cache with an embedding function, a data manager for storage and vectors, and a similarity evaluator, then calling the LLM through its adapter. The quick start takes a few lines.

The documentation’s configuration example pairs SQLite storage with a FAISS vector store and ONNX embeddings, and uses SearchDistanceEvaluation as the evaluator. With the default settings, GPTCache runs as an exact-match cache; enabling embeddings turns on semantic matching. GPTCache can also run as a server with caching and conversation support.

from gptcache import cache
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

onnx = Onnx()
data_manager = get_data_manager(CacheBase("sqlite"), VectorBase("faiss", dimension=onnx.dimension))
cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(),
)
cache.set_openai_key()
# Then call the model through gptcache.adapter.openai instead of the openai package.

This follows the structure in GPTCache’s own documentation. Check the current README before use, because the last PyPI release, 0.1.44, is from August 2024 and model provider APIs have changed since then.

Can GPTCache use Weaviate?

GPTCache can use Weaviate as its vector store, since Weaviate is among the vector stores its README lists. You can also build the same cache directly on Weaviate without GPTCache.

Weaviate showed the semantic cache idea in its Healthsearch demo, where queries like “Helpful for joint pain” and “Good for joint pain” reuse one generated result. Building the cache directly on Weaviate gives you features GPTCache leaves to other modules: filters to scope hits by tenant or prompt version, and object time-to-live to expire entries, available since Weaviate 1.36.

The direct approach needs a few dozen lines: store the prompt, response, and scope fields; query with a strict distance limit and filters; insert on a miss. Semantic caching for LLM apps shows that code, and semantic cache invalidation covers expiry.

Is GPTCache still maintained?

GPTCache’s maintenance is mixed: its last PyPI release, 0.1.44, came out in August 2024, but the GitHub repository received new commits in September 2026. Check the repository before depending on it.

PyPI shows a steady release cadence from version 0.1.1 in April 2023 through 0.1.43 in November 2023, then 0.1.44 in August 2024. On GitHub, the repository is not archived, has about 8,200 stars and around 100 open issues, and its most recent commits added cache-reuse benchmark work in September 2026. That suggests renewed activity without a packaged release yet.

For production, the practical question is whether you can fix or replace a module if needed. GPTCache’s modular design helps there, since each piece is small. Teams that already run a vector database often build the cache directly instead. AI gateway semantic caching covers gateways that offer caching as a feature.

What are the alternatives to GPTCache?

Alternatives to GPTCache include building the cache directly on a vector database such as Weaviate, Redis-based semantic caching, framework caches in LangChain, AI gateways with built-in caching, and provider prompt caching. They differ in control and effort.

Provider prompt caching is a different technique: it lowers the cost of a call with a repeated prefix but does not skip the call. Semantic caching versus prompt caching compares the two. Whichever option you choose, the evaluator problem remains: tune the match decision on pairs from your own traffic.

How do you measure a GPTCache deployment?

You measure a GPTCache deployment with the three metrics its README names, hit ratio, latency, and recall, plus a wrong-answer rate on sampled hits. The last is the one most teams forget.

Hit ratio is the share of requests served from the cache. Latency is the time to check the cache and return a result. GPTCache defines recall as the share of requests served by the cache out of those that should have been. The repository includes a sample benchmark using SQLite, FAISS, and ONNX to start from.

Add a precision check: sample cached hits regularly and confirm each cached answer fits the new question. Our results suggest an untuned evaluator will either miss most paraphrases or return wrong answers for look-alikes, so the precision check decides whether the cache is safe to keep on.

GPTCache is a modular, MIT-licensed library that puts a semantic cache in front of LLM calls: embed the prompt, search a vector store, let an evaluator decide, and return or store the answer. It supports many embedding models and vector stores, including Weaviate. Its weak point is the one every semantic cache shares: in our evaluator test, neither cosine distance nor an off-the-shelf duplicate model told paraphrases from look-alike questions. Check the repository’s current state, tune the evaluator on your own pairs, and measure wrong answers before relying on it.