PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

TL;DR

PyramidKV introduces a layer-wise dynamic KV cache compression method based on pyramidal information funneling in LLMs, significantly reducing memory while maintaining or improving long-context processing accuracy.

Contribution

This paper presents PyramidKV, a novel dynamic KV cache compression technique that allocates cache resources based on layer importance, inspired by information aggregation patterns in LLMs.

Findings

PyramidKV matches full cache performance with only 12% cache usage.

At 0.7% cache, PyramidKV outperforms other methods with up to 20.5% accuracy gain.

128 KV entries suffice for LLAMA-3-70B to achieve perfect accuracy.

Abstract

In this study, we investigate whether attention-based information flow inside large language models (LLMs) is aggregated through noticeable patterns for long context processing. Our observations reveal that LLMs aggregate information through Pyramidal Information Funneling where attention is scattering widely in lower layers, progressively consolidating within specific contexts, and ultimately focusing on critical tokens (a.k.a massive activation or attention sink) in higher layers. Motivated by these insights, we developed PyramidKV, a novel and effective KV cache compression method. This approach dynamically adjusts the KV cache size across different layers, allocating more cache in lower layers and less in higher ones, diverging from traditional methods that maintain a uniform KV cache size. Our experimental evaluations, utilizing the LongBench benchmark, show that PyramidKV matches the performance of models with a full KV cache while retaining only 12% of the KV cache, thus significantly reducing memory usage. In scenarios emphasizing memory efficiency, where only 0.7% of the KV cache is maintained, PyramidKV surpasses other KV cache compression techniques, achieving up to a 20.5 absolute accuracy improvement on TREC dataset. In the Needle-in-a-Haystack experiment, PyramidKV outperforms competing methods in maintaining long-context comprehension in LLMs; notably, retaining just 128 KV cache entries enables the LLAMA-3-70B model to achieve 100.0 Acc. performance.