SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression

TL;DR

SWAN is a novel, decompression-free KV-cache compression method for LLMs that significantly reduces memory usage during inference while maintaining near-baseline performance, offering runtime-tunable compression levels.

Contribution

SWAN introduces a fine-tuning-free, orthogonal matrix-based approach for direct KV-cache compression, enabling dynamic memory reduction without information loss or decompression overhead.

Findings

Maintains performance close to uncompressed models at 50-60% memory savings.

Offers runtime-tunable compression levels for flexible memory management.

Outperforms existing compression methods in efficiency and flexibility.

Abstract

Large Language Models (LLMs) face a significant bottleneck during autoregressive inference due to the massive memory footprint of the Key-Value (KV) cache. Existing compression techniques like token eviction, quantization, or other low-rank methods often risk information loss, have fixed limits, or introduce significant computational overhead from explicit decompression steps. In this work, we introduce SWAN, a novel, fine-tuning-free framework that eliminates this overhead. Our method uses an offline orthogonal matrix to rotate and prune the KV-cache, which is then used directly in the attention computation without any reconstruction. Our extensive experiments demonstrate that SWAN, augmented with a small dense buffer, offers a robust trade-off, maintaining performance close to the uncompressed baseline even at aggressive 50-60% memory savings per-token on KV-cache. A key advantage is its runtime-tunable compression level, allowing operators to dynamically adjust the memory footprint, a flexibility absent in methods requiring fixed offline configurations. This combination of a decompression-free design, high performance under compression, and adaptability makes SWAN a practical and efficient solution for serving LLMs with long contexts.