Inference-Time Hyper-Scaling with KV Cache Compression

TL;DR

This paper introduces a novel method called Dynamic Memory Sparsification (DMS) for compressing KV caches in Transformer LLMs, enabling inference-time hyper-scaling that improves accuracy without increasing compute or memory load.

Contribution

The paper proposes DMS, a new KV cache compression technique that maintains accuracy at high compression ratios and demonstrates its effectiveness across multiple LLMs and tasks.

Findings

DMS achieves 8× compression with only 1K training steps.

DMS improves accuracy on multiple benchmarks for scaled inference.

Inference-time hyper-scaling boosts LLM performance without additional latency.

Abstract

Inference-time scaling trades efficiency for increased reasoning accuracy by generating longer or more parallel sequences. However, in Transformer LLMs, generation cost is bottlenecked by the size of the key-value (KV) cache, rather than the number of generated tokens. Hence, we explore inference-time hyper-scaling: by compressing the KV cache, we can generate more tokens within the same compute budget and further improve the accuracy of scaled inference. The success of this approach, however, hinges on the ability of compression methods to preserve accuracy even at high compression ratios. To make hyper-scaling practical, we introduce Dynamic Memory Sparsification (DMS), a novel method for sparsifying KV caches that only requires 1K training steps to achieve 8$\times$ compression, while maintaining better accuracy than training-free sparse attention. Instead of prematurely discarding cached tokens, DMS delays token eviction, implicitly merging representations and preserving critical information. We demonstrate the effectiveness of inference-time hyper-scaling with DMS on multiple families of LLMs, showing that it boosts accuracy for comparable inference latency and memory load. For instance, we enhance Qwen-R1 32B by 12.0 points on AIME 24, 8.6 on GPQA, and 9.7 on LiveCodeBench on average for an equivalent number of memory reads.