Semantic Caching
A cache key that means “close enough”
A conventional cache answers an exact question: does this key equal a stored key? A semantic cache embeds the prompt, searches previous prompt vectors, and reuses an answer when the nearest match crosses a similarity threshold. “How do I reset my password?” can hit “I forgot my password,” even though the strings differ.
That flexibility is also the danger. Similarity is not equivalence. “Cancel my order” and “Can I cancel my order?” are close in embedding space but may require different actions. A false cache hit is not a miss or a timeout—it is a fast, cheap, confident wrong answer.
The cache belongs around workloads with repeatable, stable answers: support FAQs, documentation questions, classification, and deterministic transformations. It is a poor default for personalised, rapidly changing, permission-sensitive, or side-effecting responses.
The threshold is a product-quality control, not merely a performance knob. It should be selected from labelled query pairs and monitored like a model decision boundary.