Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation

TL;DR

This paper introduces a learning-based framework for semantic caching in large language model serving, optimizing cache eviction strategies under uncertainty to reduce inference costs effectively.

Contribution

It develops a principled, theoretically grounded approach for semantic cache eviction, including offline and online algorithms with proven guarantees, addressing real-world uncertainties.

Findings

Algorithms achieve state-of-the-art performance in synthetic tests.

Proven efficiency and adaptability of the proposed methods.

Effective handling of unknown query and cost distributions.

Abstract

Large Language Models (LLMs) are revolutionizing how users interact with information systems, yet their high inference cost poses serious scalability and sustainability challenges. Caching inference responses, allowing them to be retrieved without another forward pass through the LLM, has emerged as one possible solution. Traditional exact-match caching, however, overlooks the semantic similarity between queries, leading to unnecessary recomputation. Semantic caching addresses this by retrieving responses based on semantic similarity, but introduces a fundamentally different cache eviction problem: one must account for mismatch costs between incoming queries and cached responses. Moreover, key system parameters, such as query arrival probabilities and serving costs, are often unknown and must be learned over time. Existing semantic caching methods are largely ad-hoc, lacking theoretical foundations and unable to adapt to real-world uncertainty. In this paper, we present a principled, learning-based framework for semantic cache eviction under unknown query and cost distributions. We formulate both offline optimization and online learning variants of the problem, and develop provably efficient algorithms with state-of-the-art guarantees. We also evaluate our framework on a synthetic dataset, showing that our proposed algorithms perform matching or superior performance compared with baselines.