Crystal-KV: Efficient KV Cache Management for Chain-of-Thought LLMs via Answer-First Principle

TL;DR

Crystal-KV introduces an answer-first principle and an attention-based cache management algorithm to optimize KV cache usage in Chain-of-Thought reasoning, significantly enhancing efficiency without sacrificing accuracy.

Contribution

It proposes a novel cache management framework tailored for CoT reasoning, including an attention-based eviction strategy and adaptive cache allocation, improving throughput and response time.

Findings

Achieves state-of-the-art KV cache compression

Significantly improves inference throughput and speed

Maintains or improves answer accuracy in CoT tasks

Abstract

Chain-of-Thought (CoT) reasoning in large language models (LLMs) significantly improves accuracy on complex tasks, yet incurs excessive memory overhead due to the long think-stage sequences stored in the Key-Value (KV) cache. Unlike traditional generation tasks where all tokens are uniformly important, CoT emphasizes the final answer, rendering conventional KV compression strategies ineffective. In this paper, we present Crystal-KV, an efficient KV cache management framework tailored for CoT reasoning. Our key insight is the answer-first principle. By mapping answer preferences into think-stage attention map, we distinguish between SlipKV, which mainly maintains the reasoning flow but may occasionally introduce misleading context, and CrystalKV, which truly contributes to the correctness of the final answer. Next, we propose an attention-based Least Recently Frequently Used algorithm. It precisely identifies when a SlipKV entry's utility expires and evicts it, retaining CrystalKV without disrupting reasoning flow. Finally, we introduce an adaptive cache budget allocation algorithm. Based on the dynamic proportion of CrystalKV, it estimates the importance of each layer/head and adjusts the KV cache budget during inference, amplifying critical components to improve budget utilization. Results show that Crystal-KV achieves state-of-the-art KV cache compression, significantly improves throughput, and enables faster response time, while maintaining, or even improving, answer accuracy for CoT reasoning.