Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query

TL;DR

This paper introduces Lookahead Q-Cache, a new KV cache eviction method for large language models that uses pseudo lookahead queries to improve eviction accuracy and consistency, leading to better performance under limited memory.

Contribution

LAQ is a novel eviction framework that generates low-cost pseudo lookahead queries to better align cache eviction with actual inference queries, improving efficiency.

Findings

LAQ outperforms existing methods on LongBench and Needle-in-a-Haystack benchmarks.

LAQ achieves 1-4 points improvement under limited cache budgets.

LAQ can be combined with other approaches for further gains.

Abstract

Large language models (LLMs) rely on key-value cache (KV cache) to accelerate decoding by reducing redundant computations. However, the KV cache memory usage grows substantially with longer text sequences, posing challenges for efficient deployment. Existing KV cache eviction methods prune tokens using prefilling-stage attention scores, causing inconsistency with actual inference queries, especially under tight memory budgets. In this paper, we propose Lookahead Q-Cache (LAQ), a novel eviction framework that generates low-cost pseudo lookahead queries to better approximate the true decoding-stage queries. By using these lookahead queries as the observation window for importance estimation, LAQ achieves more consistent and accurate KV cache eviction aligned with real inference scenarios. Experimental results on LongBench and Needle-in-a-Haystack benchmarks show that LAQ outperforms existing methods across various budget levels, achieving a 1 $\sim$ 4 point improvement on LongBench under limited cache budget. Moreover, LAQ is complementary to existing approaches and can be flexibly combined to yield further improvements.