CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference

TL;DR

CHESS is a novel system that improves long-context LLM inference by dynamically managing KV cache with a context-aware hierarchical approach, achieving high throughput and quality with minimal cache usage.

Contribution

It introduces a combined algorithm-system design for KV-cache management, enabling efficient, high-quality long-context inference with reduced cache and latency.

Findings

Surpasses full KV quality with only 1% cache

Achieves up to 4.56× throughput increase

Outperforms existing baselines in stability and speed

Abstract

Long-context LLMs demand accurate inference at low latency, yet decoding becomes primarily constrained by KV cache as context grows. Prior pruning methods are largely context-agnostic: their token selection ignores step-wise relevance and local semantics, which undermines quality. Moreover, their irregular accesses and selection overheads yield only limited wall-clock speedups. To address this, we propose \textbf{CHESS}, an \textit{algorithm-system co-design} KV-cache management system. Algorithmically, CHESS introduces a context-aware, hierarchical selection policy that dynamically reconstructs a coherent context for the current decoding. System-wise, coarse granularity selection eliminates expensive data movement, fully realizing practical acceleration from theoretical sparsity. Extensive evaluations demonstrate that CHESS surpasses Full-KV quality using only \textbf{1\%} of the KV cache, delivers low-latency stable inference with up to \textbf{4.56$\times$} higher throughput, and consistently outperforms other strong baselines. Code is available at \href{https://anonymous.4open.science/r/CHESS-9958/}{https://anonymous.4open.science/r/CHESS/}.