PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference

TL;DR

PyramidInfer is a novel method that compresses the KV cache in large language models by layer-wise retaining crucial context, significantly reducing memory usage and increasing inference throughput without performance loss.

Contribution

It introduces a layer-wise KV cache compression technique based on attention weight consistency, addressing inter-layer dependencies and pre-computation memory issues.

Findings

2.2x throughput improvement over baseline

54% GPU memory reduction in KV cache

Maintains model performance with reduced memory usage

Abstract

Large Language Models (LLMs) have shown remarkable comprehension abilities but face challenges in GPU memory usage during inference, hindering their scalability for real-time applications like chatbots. To accelerate inference, we store computed keys and values (KV cache) in the GPU memory. Existing methods study the KV cache compression to reduce memory by pruning the pre-computed KV cache. However, they neglect the inter-layer dependency between layers and huge memory consumption in pre-computation. To explore these deficiencies, we find that the number of crucial keys and values that influence future generations decreases layer by layer and we can extract them by the consistency in attention weights. Based on the findings, we propose PyramidInfer, a method that compresses the KV cache by layer-wise retaining crucial context. PyramidInfer saves significant memory by computing fewer keys and values without sacrificing performance. Experimental results show PyramidInfer improves 2.2x throughput compared to Accelerate with over 54% GPU memory reduction in KV cache.