Semantic Caching

What is semantic caching?

Semantic caching stores LLM responses keyed by the embedding of the request. When a new query arrives, the cache checks for a previous query within a similarity threshold — “how do I reset my password?” and “forgot my password, help” can return the same cached answer with zero model calls. Exact-match caching catches almost nothing in natural language; semantic matching is what makes caching LLM traffic viable.

Don’t confuse the three caches

• Semantic cache — skips the model entirely for similar requests (this page).
• Prompt / KV cache — provider-side discount for repeated prompt prefixes; the model still runs.
• CDN/HTTP cache — identical bytes only; useless for chat.

They stack: semantic cache first, prompt caching on misses.

When it works — and when it’s dangerous

Great fit: high-volume support Q&A, documentation lookup, FAQ-shaped traffic where many users ask the same things (hit rates of 20–60% are realistic). Poor fit: personalized answers, anything time-sensitive, and multi-turn context — a cached answer that ignores the conversation so far reads as broken. The threshold is the whole game: too loose and users get answers to adjacent questions (worse than slow — wrong); too tight and the hit rate collapses. Tune it against a labeled set like any retrieval system (evals).

Cost reality

A cache hit costs an embedding lookup (~fractions of a cent, tens of ms) versus a full generation (cents-to-dollars, seconds). At FAQ-heavy scale, 30% hit rate ≈ 30% off the inference bill plus dramatically better p50 latency. Budget for invalidation from day one: when the underlying docs change, stale cached answers become confidently outdated.

What people get wrong

• Caching per-user content globally. Leaking one user’s cached answer to another is a privacy incident, not a bug.
• No TTL or invalidation story. Semantic caches serve yesterday’s truth by design; tie expiry to content updates.
• Measuring hit rate but not hit quality. Sample cache hits and grade whether the served answer actually fits the new query.

Try it in code

A working semantic cache in one class — threshold is the knob that matters:

import numpy as np

class SemanticCache:
    def __init__(self, threshold=0.92):
        self.threshold = threshold
        self.entries = []                     # [(vector, response)]

    def get(self, query_vec):
        for vec, response in self.entries:
            sim = float(np.dot(query_vec, vec) /
                        (np.linalg.norm(query_vec) * np.linalg.norm(vec)))
            if sim >= self.threshold:
                return response               # cache hit - no model call
        return None

    def put(self, query_vec, response):
        self.entries.append((query_vec, response))

cache = SemanticCache()
def cached_llm(query):
    qv = np.array(embed(query))
    hit = cache.get(qv)
    if hit is not None:
        return hit
    response = llm(query)
    cache.put(qv, response)
    return response

Sample your cache hits regularly and grade whether the served answer truly fits the new query — hit rate without hit quality is a vanity metric.