$A^3$: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Serving

TL;DR

The paper introduces A^3, an attention-aware method for more accurate and efficient KV cache fusion in large language model serving, significantly reducing latency and improving task performance.

Contribution

It proposes the A^3 algorithm that selectively fuses KV caches based on relevance, addressing misalignment issues in previous reuse methods for better accuracy and efficiency.

Findings

A^3 outperforms four baseline methods in task performance.

A^3 reduces time-to-first-token by 2 times.

Extensive experiments validate the effectiveness of A^3 across benchmarks and models.

Abstract

Large language models (LLMs) have demonstrated strong capabilities in processing long contexts, enabling them to tackle tasks involving long textual inputs such as multi-turn conversations, legal documents, or retrieved documents in Retrieval-Augmented Generation (RAG) systems. However, despite their ability to handle long sequences, the resulting decoding latency and memory overhead remain substantial, posing challenges for real-world deployment. Recent advances in KV Cache reuse have shown potential to mitigate these costs, but still suffer from notable performance degradation. To address this issue, we conduct an in-depth investigation of recomputation-based reuse methods and observe that the recomputed tokens often fail to align with the context segments most relevant to the question. This misalignment hinders proper updates to the critical contextual representations. Therefore, we propose the $\textbf{A}$ttention-$\textbf{A}$ware $\textbf{A}$ccurate KV Cache Fusion algorithm ($A^3$), which precomputes and selectively fuses the KV Cache of text chunks based on their relevance to the question, achieving accurate integration with minimal computational overhead. Extensive experiments on various benchmarks and LLMs demonstrate that $A^3$ achieves the best task performance compared to four baselines while reducing the time-to-first-token (TTFT) by 2$\times$.