FDC: Fast KV Dimensionality Compression for Efficient LLM Inference

TL;DR

FDC is a novel system that accelerates large-language model inference by efficiently compressing key-value pairs without decompression overhead, adapting compression rates dynamically, and balancing workloads to reduce latency and improve throughput.

Contribution

FDC introduces a fast, adaptive KV dimensionality compression method that eliminates decompression overhead and enhances inference efficiency in large-language models.

Findings

Reduces Job Completion Time by up to 64%

Achieves up to 1.97X throughput at the same latency

Maintains 99% of original accuracy without compression

Abstract

In large-language models, memory constraints in the Key-Value Cache (KVC) pose a challenge during inference. In this work, we propose FDC, a fast KV dimensionality compression system that eliminates the decompression overhead incurred in the existing KV dimensionality compression system, Palu, and reduces attention time. Moreover, FDC employs adaptive compression, tailoring KV compression rates across heads and layers based on their contributions to inference to maximize overall compression while maintaining an accuracy loss constraint. Additionally, FDC enhances the attention kernel to balance the uneven workloads caused by the adaptive compression approach to further reduce attention computation latency. Comprehensive experiments demonstrate that compared to Palu, FDC can reduce Job Completion Time (JCT) by up to 64%, and delivers up to 1.97X throughput under the same latency, while maintaining 99% of the accuracy without compression. When state-of-the-art eviction and quantization methods are combined with FDC, they exhibit similar improvements compared to those combined with Palu. We open-sourced the code.