Semantic cache: when to expire and invalidate

By Ivo Donker — compiled with AI assistance (Claude & Gemini) · Last updated: August 7, 2026

In production AI applications, latency and API costs are two of the biggest obstacles to scaling. Adding a caching layer is therefore often one of the first optimizations developers make. As discussed in the guide to caching LLM responses, a smart cache can cut response time from several seconds to a few milliseconds while saving the large language model's (LLM) tokens entirely.

Traditional key-value caches fall short with natural language, however. Questions such as "How do I change my password?" and "Changing my password, where do I do that?" have exactly the same intent but produce completely different hashes. This is where a semantic cache comes in. But while storing answers on the basis of meaning is relatively straightforward, managing the lifecycle of that data — specifically expiry (eviction) and invalidation — brings complex challenges with it.

In this article we dive deep into the strategies for expiring semantic cache entries in time and invalidating them explicitly. We cover the delicate balance of similarity thresholds, the difference between space management and correctness management, namespacing against data leaks, and how to measure your cache quality.

Exact-key cache versus semantic cache

To understand why invalidation is so challenging in a semantic cache, we first have to look at the fundamental difference with a traditional exact-key cache.

In a exact-key cache (Redis or Memcached based on a SHA-256 hash, for instance) the relationship between key and value is 1-to-1 and fully deterministic. Request exactly the same string and you get a hit. Change a single punctuation mark or space and it is a miss. Invalidation is trivial here: when a source changes, you delete the specific key cache:user_123:profile from memory.

In a semantic cache we convert the incoming prompt into a vector — a multidimensional series of numbers — using an embedding model. We then search a vector index for existing prompts with a high cosine similarity or a low Euclidean distance. For the basics of vector search, see the explanation of semantic search.

Property	Exact-key cache (hash-based)	Semantic cache (vector-based)
Linkage	Exact string match (1-to-1)	Similarity of meaning in vector space (1-to-n region)
Lookup time	O(1) — constant time	O(log N) to O(N) — approximate nearest neighbor (ANN)
Invalidation impact	Targeted: delete exactly one known key	Regional: an entire cloud of related vectors expires
Risk of errors	Zero (provided the hash is unique)	False positives (wrong hit at a low threshold)

The crucial difference for invalidation is that a change at the source does not simply invalidate one key but a region in vector space. If the returns policy changes from "14 days" to "30 days", hundreds of variations of questions about returns are suddenly out of date, regardless of the exact wording in which they were once stored.

The threshold dilemma: cosine similarity and the balancing act

The heart of a semantic cache is the similarity threshold, usually measured in cosine similarity (a scale of 0.0 to 1.0). The choice of threshold determines the balance between hit rate and the precision of the answer.

1. A threshold set too low (e.g. < 0.88 with standard models)

Suppose you set the threshold too low in order to save as many LLM calls as possible. The cache sees the question "How can I cancel my subscription?" and compares it with the cached question "Can I pause my subscription temporarily?". At too low a threshold, the vector index matches these questions because both concern "stopping or changing a subscription". The user who wants to cancel now gets the instructions for pausing. That is a critical false positive hit.

2. A threshold set too high (e.g. > 0.97)

Set the threshold extremely high to avoid errors and you lose the power of semantics. Subtle synonyms or typos (such as "reset password" versus "restore password") fall just short of the threshold. The cache generates constant misses, so you still have to call the expensive LLM for every variant.

Eviction versus invalidation: two different concepts

In practice the terms eviction (clearing out) and invalidation (marking as invalid) are often used interchangeably. A robust architecture requires them to be kept strictly apart.

• Eviction (space management): Removing items from the cache to keep memory or storage capacity from filling up. Eviction is a capacity question. A cached question may still be 100% correct in substance, yet it is removed because room is needed for more popular data.
• Invalidation (correctness management): Explicitly marking or removing items because the data has become incorrect, out of date or unsafe . Invalidation happens independently of available storage.

Eviction strategies for semantic caches

Because vector indexes load index structures (such as HNSW graphs) on top of RAM, smart eviction is essential:

1. LRU (least recently used): Removes the vectors that have gone unreferenced the longest. Excellent for general support questions with clear trends.
2. LFU (least frequently used): Removes questions that are asked least often over a longer period. Prevents a one-off spike in a specific question from clogging memory.
3. TTL (time to live): A preset maximum lifetime (24 hours, for instance). TTL is a passive form of eviction, but it is unfortunately abused far too often as a poor substitute for real invalidation.
4. Cost-based eviction: A strategy unique to LLM caches. Instead of treating all items equally, you assign a weight based on the cost of the original generated response. An answer produced by a heavy agent with 10 RAG searches and 4,000 output tokens is expensive to recompute. You keep that answer in the cache longer than a simple 50-token answer from a lightweight model.

Invalidation triggers in AI systems

Real invalidation is triggered by events in the application landscape. A semantic cache has to listen for the following five specific triggers:

1. Source content changed (RAG & data sources)

In retrieval-augmented generation (RAG) systems, the LLM's answer leans on external documents or databases. When a source document is updated or deleted, all related cached answers are potentially invalid. To solve this, link the vectors in the cache to the source IDs of the RAG documents. For handling index changes, see also the article on vector index maintenance.

2. Prompt version or system prompt changed

When developers adjust the system prompt — to change the tone, add formatting requirements or tighten safety instructions — old cached answers no longer meet the new specification. How to organize this version control is covered in version control for prompts in code.

3. Model version or model parameters changed

An upgrade from gpt-4o-2024-05-13 to a newer snapshot, or an adjustment to parameters such as temperature or top_p, changes the expected output of the pipeline. A change in model configuration means old answers must no longer be served.

4. User permissions and authorization changed

When a user is promoted from a "standard" to an "admin" role, they may be entitled to more extensive answers. Cache entries generated under limited permissions must not simply be returned to a user with higher (or lower) authorization.

5. Time-bound or dynamic facts

Questions containing time-dependent words (such as "today", "current quarterly figures", "who is the current CEO") carry a built-in expiry date. If the prompt is time-sensitive, the cache entry must be given a very short TTL or be filtered out explicitly by an entity recognizer before storage.

Cache key design and deterministic invalidation

To avoid having to wipe the entire vector database by hand on every prompt or parameter change, apply hybrid cache keys . Here you combine deterministic metadata with the semantic vector.

A robust semantic cache entry consists of two parts:

1. The vector: Computed from the cleaned user prompt and nothing else.
2. The metadata filter payload: A strict combination of environment variables.

Before the semantic similarity (cosine similarity) is computed, the vector database first performs exact metadata filtering. If there is no match on the metadata, the item is skipped, however close the vectors may be.

// Voorbeeld van een hybride cache-lookup structuur
{
  "vector": [0.012, -0.045, 0.389, ...], // Embedding van de gebruikersvraag
  "filter": {
    "tenant_id": "org_9871",
    "prompt_hash": "a1b2c3d4e5",          // SHA256 van system prompt + sjabloon
    "model_name": "gpt-4o",
    "temperature": 0.0,
    "user_role": "finance_manager"
  }
}

The advantage: Adjust the system prompt and the prompt_hashchanges. On the very next lookup, the metadata filter no longer matches. The old cache entries are thereby de facto invalidated immediately, without any expensive write or delete operation in the vector index. Old entries then expire by themselves through the regular LRU eviction process.

Multi-tenancy and security: preventing data leaks

When implementing a semantic cache in environments with multiple users or organizations, the following rules must be applied without exception:

• Hard namespace separation: Every tenant gets its own isolated vector namespace or a mandatory hard metadata filter (tenant_id = X) that the backend code enforces inescapably. For architectural patterns, see the article on multi-tenant LLM applications.
• No caching of personal data (PII): Does a prompt or answer contain personal data, specific account numbers or unique IDs? Then simply do not store the answer in the semantic cache. Use anonymization tools to strip PII before processing if caching really is necessary.
• Permission-bound namespacing: Make the authorization role (RBAC) part of the filter context. An answer generated for an administrator must never be served to a standard user.

Handling stale embeddings when models change

An often overlooked pitfall in semantic caching is updating the embedding model itself (switching from OpenAI's text-embedding-ada-002 to text-embedding-3-large or an open-source BGE model, for instance).

Embedding vectors from different models live in entirely different mathematical spaces. A vector computed with model A cannot be compared with a vector computed with model B. Even if both models happen to share the same dimensionality (1536, say), the distances between them are meaningless.

When you move to a new embedding model, you have two options:

1. Complete cache invalidation (cold flush): You wipe the entire vector cache and rebuild it from scratch with the new embedding model. This temporarily lowers the hit rate and raises LLM costs, but it is simple and clean to implement.
2. Blue-green cache deployment (dual writing): You set up a new vector index alongside the old one. New questions are embedded with the new model and stored in the new index. Lookups consult the new cache first. After a transition period (7 days, for instance) you detach the old index and delete it.

Measuring cache quality: KPIs and monitoring

You can only manage a semantic cache effectively once you measure its performance quantitatively. A high hit rate looks positive, but if the share of wrong hits rises, user satisfaction drops sharply. For detailed cost calculations, see the guide to monitoring LLM costs.

Steer your semantic cache on the following four core KPIs:

1. Cache hit rate:
   Hit Rate = (Aantal Cache Hits / Totaal Aantal Vragen) * 100%
   A healthy semantic cache in a customer service environment typically achieves a hit rate between 30% and 60%.
2. False positive rate:
   The share of cases in which the cache returns a hit but the cached answer does not match the new question in substance. You measure this by having a periodic sample (1% of all hits, say) assessed by a powerful LLM (LLM-as-a-judge) or human evaluators. The target figure is < 0.5%.
3. Latency reduction:
   The average time saved per request. Typically the P95 response time falls from 2,500ms (an LLM call) to < 50ms (vector search plus cache return).
4. Cost saving per day/month:
   The calculated value of prompt and completion tokens saved at the LLM provider, minus the operating costs of the embedding model and the vector database.

Common mistakes and how to avoid them

Designing an invalidation strategy for semantic caches, we see the same mistakes recur in practice:

1. Using TTL as a substitute for invalidation: Setting a TTL of 7 days and hoping that outdated information "will disappear by itself". If a product price changes immediately, the cache serves incorrect information for 7 days. Use event-driven invalidation for dynamic data.
2. Forgetting the system prompt in the cache key: Changing your chatbot's instructions and then noticing that users still get answers in the old style, because the cache matches on the user prompt alone.
3. Not monitoring the threshold: Setting a threshold to 0.90 once and never looking at it again. As the dataset and the type of questions change, the optimal threshold can shift.
4. Applying caching to non-deterministic or creative tasks: Trying to cache prompts such as "Write a unique poem about..." or "Generate a random test dataset". Caching is meant for informative, consistent questions, not for creative generation.

Conclusion

A semantic cache is a powerful instrument for speeding up LLM applications and cutting infrastructure costs sharply. Its success stands or falls, however, with how the data lifecycle is managed.

By drawing a sharp line between spatial eviction (LRU/LFU/cost) and substantive invalidation (events, metadata filters and version hashes), you prevent users from receiving outdated or incorrect information. Keep tenants strictly separated to rule out data leaks, and measure the false positive rate continuously to safeguard the balance between speed and precision.