FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension

TL;DR

FreqKV introduces a frequency domain-based, parameter-free method for compressing key-value caches in large language models, significantly extending context windows while preserving performance.

Contribution

It proposes a novel, architecture-agnostic approach that compresses KV caches in the frequency domain, enabling efficient processing of ultra-long contexts without retraining.

Findings

Extends LLaMA-2's context window to 256K tokens.

Outperforms existing KV cache compression methods.

Maintains stable perplexity on long-context benchmarks.

Abstract

Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local information in both long prefilling and decoding scenarios. When extrapolating beyond the pretrained context length, their performance degrades sharply on long-context benchmarks. Motivated by the observation in the frequency domain that the context information is concentrated in the low-frequency components, we propose FreqKV, a parameter-free and architecture-agnostic approach. It iteratively compresses the increasing KV cache in the frequency domain, allowing models to process lengthy contexts efficiently. With minimal training at 8K length, FreqKV extends the context window of LLaMA-2-7B up to 256K tokens while maintaining stable perplexity. Extensive experiments across prefilling and decoding demonstrate that FreqKV enables robust context window extension and consistently outperforms existing KV cache compression methods on LLaMA-2 and LLaMA-3, highlighting its effectiveness for both understanding and generation in long contexts.