FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference

TL;DR

FreeKV is a training-free framework that significantly improves KV cache retrieval efficiency for large language models, achieving up to 13 times speedup while maintaining near-lossless accuracy across various models and scenarios.

Contribution

FreeKV introduces speculative retrieval and hybrid KV layouts, enabling efficient, accurate, and training-free KV cache management for LLM inference.

Findings

Achieves up to 13× speedup over state-of-the-art methods.

Maintains near-lossless accuracy across multiple models.

Effectively overlaps retrieval with computation for latency hiding.

Abstract

Large language models (LLMs) are widely deployed with rapidly expanding context windows to support increasingly demanding applications. However, long contexts pose significant deployment challenges, primarily due to the KV cache whose size grows proportionally with context length. While KV cache compression methods have been proposed to address this issue, KV dropping methods incur considerable accuracy loss, and KV retrieval methods suffer from significant efficiency bottlenecks. We propose FreeKV, a training-free algorithm-system co-optimization framework to enhance KV retrieval efficiency while preserving accuracy. On the algorithm side, FreeKV introduces speculative retrieval to shift the KV selection and recall processes out of the critical path, combined with fine-grained correction to ensure accuracy. On the system side, FreeKV employs hybrid KV layouts across CPU and GPU memory to eliminate fragmented data transfers, and leverages double-buffered streamed recall to further improve efficiency, enabling effective overlap with computation, full latency hiding, and practical speedups from speculative recall. Experiments demonstrate that FreeKV achieves near-lossless accuracy across various scenarios and models, delivering up to a 13$\times$ speedup compared to SOTA KV retrieval methods. Code is available at https://github.com/sjtu-zhao-lab/FreeKV.