Joint Encoding of KV-Cache Blocks for Scalable LLM Serving

TL;DR

This paper introduces a joint encoding method for KV-cache blocks in large language models, significantly reducing memory usage and increasing inference throughput without hardware changes.

Contribution

It proposes a novel joint encoding technique that fuses similar KV-cache blocks, enabling scalable, high-concurrency LLM serving with theoretical and empirical validation.

Findings

Achieves up to 4.38× KV-cache compression with negligible accuracy loss.

Improves token throughput by approximately 40% in real LLM serving scenarios.

Outperforms recent compression baselines in diverse benchmarks.

Abstract

Modern large language models (LLMs) drive interactive AI systems but are bottlenecked by the memory-heavy growth of key-value (KV) caches, which limits real-time throughput under concurrent loads. Existing KV-cache compression methods rely on rigid heuristics, disrupt tensor layouts, or require specialized compute, hindering scalability and deployment. We propose joint encoding of KV-cache blocks, which fuses similar blocks across requests and input chunks into shared representations while preserving standard cache structure. This alleviates the KV-cache memory bottleneck, supporting high-concurrency serving without specialized hardware. Theoretically, we analyze the rate-distortion tradeoff of fused cache blocks under a Poisson process model. Empirically, our method achieves up to 4.38 $\times$ KV-cache compression with negligible accuracy loss across diverse LLMs and benchmarks, outperforming recent structured and adaptive compression baselines. In real LLM serving, joint encoding improves the token throughput by $\sim$40\% on a single-machine vLLM benchmark, demonstrating substantial gains in inference throughput. Code is available at https://github.com/sef1/kv_fast_fusion kv_joint_encoding.