CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

TL;DR

CacheGen is a novel system that compresses and streams KV caches for large language models, significantly reducing bandwidth and delay during context loading without sacrificing response quality.

Contribution

CacheGen introduces a custom tensor encoder and adaptive compression for KV caches, enabling faster and more bandwidth-efficient large language model serving.

Findings

Reduces KV cache size by 3.5-4.3x

Decreases total context-fetching delay by 3.2-3.7x

Maintains high generation quality

Abstract

As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging, as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays. CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, leveraging KV cache's distributional properties to encode a KV cache into more compact bitstream representations with negligible decoding overhead, to save bandwidth usage. Second, CacheGen adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth, in order to maintain low context-loading delay and high generation quality. % When available bandwidth drops, CacheGen may raise the compression level for a part of the context or recompute its KV cache on the fly. We test CacheGen on popular LLMs and datasets. Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5-4.3x and the total delay in fetching and processing contexts by 3.2-3.7x with negligible impact on the LLM response quality. Our code is at: https://github.com/UChi-JCL/CacheGen.