Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models

TL;DR

This paper presents a semantic caching method for LLM question-answering systems that significantly reduces redundant computation and maintains accuracy, improving efficiency in real-time applications.

Contribution

It introduces a novel semantic caching technique for reusing contextual summaries, reducing computational overhead in LLM-based QA workflows.

Findings

Reduces redundant computation by 50-60%.

Maintains answer accuracy comparable to full processing.

Effective on multiple datasets including NaturalQuestions and TriviaQA.

Abstract

Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing lengthy contexts in distributed systems incurs high computational overhead, memory usage, and network bandwidth. This paper introduces a novel semantic caching approach for storing and reusing intermediate contextual summaries, enabling efficient information reuse across similar queries in LLM-based QA workflows. Our method reduces redundant computations by up to 50-60% while maintaining answer accuracy comparable to full document processing, as demonstrated on NaturalQuestions, TriviaQA, and a synthetic ArXiv dataset. This approach balances computational cost and response quality, critical for real-time AI assistants.