D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models

TL;DR

D2O introduces a dynamic, discriminative KV cache compression method for large language models that reduces memory usage and improves inference speed while maintaining high-quality long-text generation without fine-tuning.

Contribution

The paper proposes D2O, a novel KV cache compression technique that dynamically and discriminatively optimizes cache retention at layer and token levels without fine-tuning.

Findings

Achieves over 3× inference throughput improvement.

Provides significant memory savings during inference.

Maintains high-quality long-text generation.

Abstract

Generative inference in Large Language Models (LLMs) is impeded by the growing memory demands of Key-Value (KV) cache, especially for longer sequences. Traditional KV cache eviction strategies, which discard less critical KV pairs based on attention scores, often degrade generation quality, leading to issues such as context loss or hallucinations. In this work, we introduce Dynamic Discriminative Operations (D2O), a KV cache compression method that optimizes KV cache size dynamically and discriminatively at two levels without fine-tuning, while preserving essential context. At layer level, D2O leverages the varying densities of attention weights between shallow and deep layers to dynamically determine which layers should avoid excessive eviction via a novel dynamic allocation strategy to minimize information loss. At token level, D2O incorporates a compensation mechanism that maintains a similarity threshold to re-discriminate the importance of currently discarded tokens, determining whether they should be recalled and merged with similar tokens. We conduct experiments on various benchmarks and LLM architectures. Our results show that D2O not only achieves significant memory savings and enhances inference throughput by more than 3$\times$ but also maintains high-quality long-text generation.