From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings

TL;DR

This paper investigates semantic caching for LLM embeddings, proposing new policies and heuristics to improve response speed and accuracy, with comprehensive evaluations demonstrating their effectiveness.

Contribution

It introduces polynomial-time heuristics for offline semantic caching and novel online policies that combine recency, frequency, and locality, advancing caching strategies for LLMs.

Findings

Frequency-based policies are strong baselines.

A novel variant improves semantic accuracy.

Effective strategies identified for current systems.

Abstract

The rapid adoption of large language models (LLMs) has created demand for faster responses and lower costs. Semantic caching, reusing semantically similar requests via their embeddings, addresses this need but breaks classic cache assumptions and raises new challenges. In this paper, we explore offline policies for semantic caching, proving that implementing an optimal offline policy is NP-hard, and propose several polynomial-time heuristics. We also present online semantic aware cache policies that combine recency, frequency, and locality. Evaluations on diverse datasets show that while frequency based policies are strong baselines, our novel variant improves semantic accuracy. Our findings reveal effective strategies for current systems and highlight substantial headroom for future innovation. All code is open source.