KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models

TL;DR

KVPruner is a structural pruning method that significantly reduces memory usage and increases throughput in large language models by selectively pruning key-value channels, with minimal fine-tuning.

Contribution

It introduces a global perplexity-based analysis for efficient KV channel pruning, achieving substantial speed and memory improvements with limited fine-tuning.

Findings

Reduces runtime memory by 50%

Increases throughput by over 35%

Requires only two hours of fine-tuning

Abstract

The bottleneck associated with the key-value(KV) cache presents a significant challenge during the inference processes of large language models. While depth pruning accelerates inference, it requires extensive recovery training, which can take up to two weeks. On the other hand, width pruning retains much of the performance but offers slight speed gains. To tackle these challenges, we propose KVPruner to improve model efficiency while maintaining performance. Our method uses global perplexity-based analysis to determine the importance ratio for each block and provides multiple strategies to prune non-essential KV channels within blocks. Compared to the original model, KVPruner reduces runtime memory usage by 50% and boosts throughput by over 35%. Additionally, our method requires only two hours of LoRA fine-tuning on small datasets to recover most of the performance.