A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference

TL;DR

This paper systematically investigates cross-layer KV sharing techniques for LLM inference, proposing a unified framework, and evaluates their efficiency and performance, revealing trade-offs in cache size reduction and sharing strategies.

Contribution

It introduces a comprehensive framework for cross-layer KV sharing methods and provides extensive experimental analysis to guide efficient LLM inference.

Findings

Reducing KV cache size by 2× can improve throughput while maintaining performance.

Further reduction favors sharing queries with upper layer KVs, increasing training cost.

Most configurations outperform standard transformers in throughput with similar performance.

Abstract

Recently, sharing key-value (KV) cache across layers has been found effective in efficient inference of large language models (LLMs). To systematically investigate different techniques of cross-layer KV sharing, we propose a unified framework that covers several recent methods and their novel variants. We conduct comprehensive experiments on all the configurations of the framework, evaluating their generation throughput and performance in language modeling and downstream tasks. We find that when reducing the size of the KV cache by 2$\times$, most configurations can achieve higher throughput than standard transformers while maintaining competitive performance. When further reducing the size of the KV cache, however, pairing queries of all layers with KVs of upper layers performs better, at the expense of additional training cost and prefilling latency. We hope that this work will help users make more informed choices of cross-layer KV sharing approaches and facilitate future research on efficient LLM inference.