Semantic caching for an LLM must return a cached answer for paraphrased prompts. What is the core mechanism?
- A Hash the prompt string and use GET
- B Embed the prompt, KNN search cached embeddings, and return the hit if distance is below a threshold (correct answer)
- C Store prompts in a Redis List and scan
- D Use SETEX with a short TTL
Shortcut: Exact-match hashing misses paraphrases. Semantic caching embeds the prompt, finds the nearest cached vector, and reuses the stored response when similarity exceeds the configured threshold.