SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging

TL;DR

SemantiCache is a novel KV cache compression method that preserves semantic integrity by clustering tokens into coherent chunks, significantly accelerating inference and reducing memory without degrading model performance.

Contribution

We propose SemantiCache, a semantic-aware compression framework using hierarchical clustering and attention rebalancing to improve cache efficiency.

Findings

Up to 2.61x faster decoding

Significant memory reduction

Maintains model performance

Abstract

Existing KV cache compression methods generally operate on discrete tokens or non-semantic chunks. However, such approaches often lead to semantic fragmentation, where linguistically coherent units are disrupted, causing irreversible information loss and degradation in model performance. To address this, we introduce SemantiCache, a novel compression framework that preserves semantic integrity by aligning the compression process with the semantic hierarchical nature of language. Specifically, we first partition the cache into semantically coherent chunks by delimiters, which are natural semantic boundaries. Within each chunk, we introduce a computationally efficient Greedy Seed-Based Clustering (GSC) algorithm to group tokens into semantic clusters. These clusters are further merged into semantic cores, enhanced by a Proportional Attention mechanism that rebalances the reduced attention contributions of the merged tokens. Extensive experiments across diverse benchmarks and models demonstrate that SemantiCache accelerates the decoding stage of inference by up to 2.61 times and substantially reduces memory footprint, while maintaining performance comparable to the original model.