ContextCache: Context-Aware Semantic Cache for Multi-Turn Queries in Large Language Models

TL;DR

ContextCache is a novel system that enhances semantic caching for multi-turn dialogues in large language models by incorporating dialogue context, leading to improved accuracy and reduced computational costs.

Contribution

It introduces a two-stage retrieval architecture with self-attention for context-aware caching in multi-turn LLM conversations, addressing limitations of previous query-only systems.

Findings

Improves cache hit precision and recall in real-world dialogues.

Reduces response latency by approximately 10 times.

Enhances computational efficiency for LLM conversational applications.

Abstract

Semantic caching significantly reduces computational costs and improves efficiency by storing and reusing large language model (LLM) responses. However, existing systems rely primarily on matching individual queries, lacking awareness of multi-turn dialogue contexts, which leads to incorrect cache hits when similar queries appear in different conversational settings. This demonstration introduces ContextCache, a context-aware semantic caching system for multi-turn dialogues. ContextCache employs a two-stage retrieval architecture that first executes vector-based retrieval on the current query to identify potential matches and then integrates current and historical dialogue representations through self-attention mechanisms for precise contextual matching. Evaluation of real-world conversations shows that ContextCache improves precision and recall compared to existing methods. Additionally, cached responses exhibit approximately 10 times lower latency than direct LLM invocation, enabling significant computational cost reductions for LLM conversational applications.