Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference

TL;DR

This paper introduces unstructured sparsity techniques for KV cache pruning in large language model inference, achieving high compression rates and faster decoding without accuracy loss.

Contribution

It presents a novel unstructured sparsity method with a custom sparse attention kernel that improves KV cache compression and decoding speed in LLMs.

Findings

Achieves up to 70% sparsity without accuracy loss.

Compresses KV cache to 45% of dense size, enabling longer contexts.

Increases decoding throughput by up to 2.23x.

Abstract

We demonstrate that unstructured sparsity significantly improves KV cache compression for LLMs, enabling sparsity levels up to 70% without compromising accuracy or requiring fine-tuning. We conduct a systematic exploration of pruning strategies and find per-token magnitude-based pruning as highly effective for both Key and Value caches under unstructured sparsity, surpassing prior structured pruning schemes. The Key cache benefits from prominent outlier elements, while the Value cache surprisingly benefits from a simple magnitude-based pruning despite its uniform distribution. KV cache size is the major bottleneck in decode performance due to high memory overhead for large context lengths. To address this, we use a bitmap-based sparse format and a custom attention kernel capable of compressing and directly computing over compressed caches pruned to arbitrary sparsity patterns, significantly accelerating memory-bound operations in decode computations and thereby compensating for the overhead of runtime pruning and compression. Our custom attention kernel coupled with the bitmap-based format delivers substantial compression of KV cache upto 45% of dense inference and thereby enables longer context length and increased tokens/sec throughput of upto 2.23x compared to dense inference. Our pruning mechanism and sparse attention kernel is available at https://github.com/dhjoo98/mustafar.