TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization

TL;DR

TailorKV is a hybrid framework that optimizes long-context inference in large language models by combining selective KV cache offloading and quantization, significantly reducing memory and latency while maintaining near-lossless performance.

Contribution

It introduces a novel hybrid compression method, TailorKV, that effectively integrates quantization and offloading based on layer-specific characteristics for improved long-context inference.

Findings

Achieves near-lossless performance with aggressive compression.

Serves 128k context on a single RTX 3090 within 82 ms per token.

Outperforms state-of-the-art methods in long-context LLM inference.

Abstract

The Key-Value (KV) cache in generative large language models (LLMs) introduces substantial memory overhead. Existing works mitigate this burden by offloading or compressing the KV cache. However, loading the entire cache incurs significant latency due to PCIe bandwidth bottlenecks in CPU-GPU communication, while aggressive compression causes notable performance degradation. We identify that certain layers in the LLM need to maintain global information and are unsuitable for selective loading. In contrast, other layers primarily focus on a few tokens with dominant activations that potentially incur substantial quantization error. This observation leads to a key insight that loading dominant tokens and quantizing all tokens can complement each other. Building on this insight, we propose a hybrid compression method, TailorKV, which seamlessly integrates quantization and offloading. TailorKV develops an inference framework along with a hardware-friendly implementation that leverages these complementary characteristics. Extensive long-context evaluations exhibit that TailorKV achieves nearly lossless performance under aggressive compression settings, outperforming the state-of-the-art. Particularly, the Llama-3.1-8B with 128k context can be served within a single RTX 3090 GPU, reaching 82 ms per token during decoding.