# KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache

## TL;DR

KVComp is a novel framework that employs specialized lossy compression techniques to significantly reduce memory usage and enhance throughput in large language model KV caches during long-text generation.

## Contribution

It introduces a generic, efficient KV cache management system with novel lossy compression tailored for LLMs, improving memory efficiency and computational performance.

## Key findings

- Achieves up to 83% memory reduction with minimal accuracy loss.
- Significantly increases inference throughput and reduces decompression overhead.
- Outperforms existing methods and accelerates matrix-vector operations.

## Abstract

Transformer-based large language models (LLMs) demonstrate impressive potential in various practical applications. However, long context inference poses a significant challenge due to the enormous memory requirements of the key-value (KV) cache, which can scale to multiple gigabytes as sequence length and batch size increase. In this paper, we present KVComp, a generic and efficient KV cache management framework optimized for long-text generation that synergistically works with both latency-critical and throughput-critical inference systems. KVComp employs novel lossy compression techniques specifically designed for KV cache data characteristics, featuring careful co-design of compression algorithms and system architecture. Our approach maintains compatibility with the growing nature of KV cache while preserving high computational efficiency. Experimental results show that KVComp achieves on average 47\% and up to 83\% higher memory reduction rate compared to existing methods with little/no model accuracy degradation. Furthermore, KVComp achieves extremely high execution throughput, effectively reducing decompression overhead and, in some cases, even accelerating the matrix-vector multiplication operation and outperform cuBLAS-based attention kernels with less data movement.