vCache: Verified Semantic Prompt Caching

TL;DR

vCache introduces a verified semantic caching system for large language models that guarantees user-defined error bounds and significantly improves cache hit rates by using an online learning algorithm to adapt similarity thresholds.

Contribution

It presents the first verified semantic cache with formal error guarantees, employing an online learning approach to optimize similarity thresholds for improved performance.

Findings

vCache outperforms static-threshold baselines with up to 12.5× higher cache hit rates.

vCache achieves up to 26× lower error rates compared to existing methods.

The system reliably meets user-defined error bounds across diverse benchmarks.

Abstract

Semantic caches return cached responses for semantically similar prompts to reduce LLM inference latency and cost. They embed cached prompts and store them alongside their response in a vector database. Embedding similarity metrics assign a numerical score to quantify the similarity between a request and its nearest neighbor prompt from the cache. Existing systems use the same static similarity threshold across all requests to determine whether two prompts can share similar responses. However, we observe that static thresholds do not give formal correctness guarantees, result in unexpected error rates, and lead to suboptimal cache hit rates. This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees for predictable performance. It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training. Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines with up to 12.5$\times$ higher cache hit and 26$\times$ lower error rates. We release the vCache implementation and four benchmarks to support future research.