InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management

TL;DR

InfiniGen introduces a dynamic KV cache management framework for large language models that reduces memory overhead and improves inference efficiency during long-text generation by selectively prefetching essential cache entries.

Contribution

The paper proposes a novel KV cache management method that enhances offloading-based LLM inference systems by selectively prefetching critical cache entries, improving performance and accuracy.

Findings

Up to 3.00x performance improvement over prior methods

Better model accuracy with efficient cache management

Effective handling of long-text generation in LLMs

Abstract

Transformer-based large language models (LLMs) demonstrate impressive performance across various natural language processing tasks. Serving LLM inference for generating long contents, however, poses a challenge due to the enormous memory footprint of the transient state, known as the key-value (KV) cache, which scales with the sequence length and batch size. In this paper, we present InfiniGen, a novel KV cache management framework tailored for long-text generation, which synergistically works with modern offloading-based inference systems. InfiniGen leverages the key insight that a few important tokens that are essential for computing the subsequent attention layer in the Transformer can be speculated by performing a minimal rehearsal with the inputs of the current layer and part of the query weight and key cache of the subsequent layer. This allows us to prefetch only the essential KV cache entries (without fetching them all), thereby mitigating the fetch overhead from the host memory in offloading-based LLM serving systems. Our evaluation on several representative LLMs shows that InfiniGen improves the overall performance of a modern offloading-based system by up to 3.00x compared to prior KV cache management methods while offering substantially better model accuracy.