QCFuse: Query-Centric Cache Fusion for Efficient RAG Inference

TL;DR

QCFuse is a cache fusion system that enhances RAG inference efficiency by globally aware token selection and selective recomputation, achieving 40% faster responses with maintained accuracy.

Contribution

It introduces a query-centric cache fusion approach using semantic anchors and attention-based token updates to improve efficiency and accuracy in LLM inference.

Findings

QCFuse improves response efficiency by 40%.

It maintains equivalent accuracy to existing methods.

In some cases, it enhances response accuracy through attention denoising.

Abstract

Cache fusion accelerates generation process of LLMs equipped with RAG through KV caching and selective token recomputation, thereby reducing computational costs and improving efficiency. However, existing methods primarily rely on local perspectives for token selection and lack global awareness from the user query. Utilizing this global awareness is challenging due to the high cost of obtaining context-aware query representations and the strict pipeline constraints required for efficient attention analysis. Thus, this demonstration introduces QCFuse, an innovative KV cache fusion system centered on the user query. QCFuse leverages semantic summary anchors to enhance query representations and selectively recomputes query-related tokens to improve accuracy, updating tokens based on the attention distribution of the most critical Transformer layer to preserve the high efficiency of the pipeline structure. Evaluations on real-world datasets demonstrate that QCFuse significantly improves the response efficiency of LLMs by 40\% while maintaining equivalent accuracy compared to current methods. Additionally, in certain scenarios, QCFuse achieves an attention denoising effect that yields higher response accuracy, demonstrating substantial potential in the optimization of LLM inference.