LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing

TL;DR

LycheeCluster introduces a structure-aware hierarchical indexing method that significantly accelerates long-context inference in large language models while maintaining performance, addressing key computational challenges.

Contribution

It proposes a novel hierarchical KV cache management technique with boundary-aware chunking and recursive indexing, improving speed and efficiency over existing methods.

Findings

Achieves up to 3.6x inference speedup

Maintains negligible performance degradation

Outperforms state-of-the-art KV cache methods

Abstract

The quadratic complexity of the attention mechanism and the substantial memory footprint of the Key-Value (KV) cache present severe computational and memory challenges for Large Language Models (LLMs) processing long contexts. Existing retrieval-based methods often compromise semantic integrity through fixed-size chunking and suffer from inefficient linear scanning. In this paper, we propose LycheeCluster, a novel method for efficient KV cache management. LycheeCluster preserves local semantic coherence via boundary-aware chunking and constructs a recursive hierarchical index rooted in the triangle inequality. This design transforms cache retrieval from a linear scan into a theoretically bounded, logarithmic-time pruning process, while a lazy update strategy supports efficient streaming generation. Experiments demonstrate that LycheeCluster achieves up to a 3.6x end-to-end inference speedup with negligible degradation in model performance, outperforming state-of-the-art KV cache management methods (e.g., Quest, ClusterKV). We will release our code and kernels after publication.