An experimental study of KV cache reuse strategies in chunk-level caching systems

TL;DR

This paper evaluates chunk-level caching strategies in retrieval-augmented generation, identifying limitations of existing methods and proposing a new combined approach that improves accuracy in large language model inference.

Contribution

The paper demonstrates fundamental limitations of current CLC methods and introduces a novel combined strategy that enhances accuracy by leveraging their complementarity.

Findings

Existing CLC approaches have fundamental limitations.

Combining CLC techniques leads to improved accuracy.

Extensive experiments validate the effectiveness of the proposed method.

Abstract

Retrieval-augmented generation improves large language models' accuracy by adding relevant retrieved text to the prompt. Chunk level caching (CLC) accelerates inference by precomputing KV caches for these retrieved chunks and reusing them. However, these caches miss cross-attention dependencies between chunks, which can reduce output quality. Several methods try to improve CLC accuracy using different techniques. We make two main contributions. First, we show that existing CLC approaches have fundamental limitations that limit their accuracy or their applicability. We back this conclusion with an extensive CLC system experimental evaluation. Second, we observe that existing CLC techniques are complementary. We leverage this insight to propose a new CLC design that carefully combines them and achieves better accuracy.