VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

TL;DR

VeriCache is an inference framework that maintains full-KV-cache output accuracy while significantly improving throughput by combining compressed KV cache drafting with verification, addressing GPU memory and swapping challenges.

Contribution

It introduces VeriCache, a novel inference method that ensures lossless output with high throughput by verifying compressed KV cache drafts against full cache without GPU memory overhead.

Findings

Achieves up to 4X higher throughput than full-KV inference.

Produces identical outputs to full-KV-cache decoding.

Supports various compression methods and integrates with existing decoding techniques.

Abstract

The large size of the KV cache has become a major bottleneck for serving LLMs with increasing context lengths. In response, many KV cache compression methods, such as token dropping and quantization, have been proposed. However, almost all of these methods are inherently lossy-despite minimal accuracy degradation for short outputs, their outputs increasingly diverge from full-KV-cache outputs as more tokens are decoded, which leads to catastrophic failures in code generation and tool calling. We present VeriCache, the first inference framework that ensures the same output as full-KV-cache decoding but largely preserves the high decoding throughput of a range of KV cache compression algorithms. VeriCache uses the compressed KV cache to draft tokens, then verifies them against the full KV cache. While it may seem like just speculative decoding, VeriCache requires addressing a key system challenge to work-keeping the full KV cache out of GPU memory and minimizing the overhead of swapping it in for verification. The insight is two-fold: (1) compressed-KV decoding can be parallelized with full-KV swap, because one is HBM-bandwidth-bound and the other is PCIe/network-bound, and (2) the compressed KV cache often produces output similar to the full KV cache, allowing a long drafting horizon to amortize each full-KV swap. VeriCache applies to both long-context decoding and remote prefix caching, supports a broad family of token-dropping and quantization methods through a uniform compressor interface, and composes with traditional speculative decoding. Experimental results show that VeriCache achieves up to 4X higher throughput than full-KV inference while producing identical outputs.