ReasonCache: Accelerating Large Reasoning Model Serving through KV Cache Sharing

TL;DR

ReasonCache is a novel KV cache management approach that leverages similarity in reasoning steps to significantly improve throughput and reduce latency in large reasoning model serving.

Contribution

It introduces a collaborative filtering-based method for efficient KV cache reuse, enhancing inference QoS without sacrificing accuracy.

Findings

Achieves up to 89.2% peak throughput improvement.

Realizes 40-60% average throughput gains.

Maintains higher accuracy compared to existing cache techniques.

Abstract

Large Reasoning Models (LRMs) are becoming integral to many AI inference systems, enhancing their capabilities with advanced reasoning. However, deploying these models in production environments presents a significant QoS challenge: the substantial memory overhead from their long, auto-regressive inference processes severely limits throughput and increases latency, thereby affecting the quality of service for concurrent users. We observe that LRMs frequently generate highly similar intermediate reasoning steps, which, in turn, correspond to highly similar KV cache states across layers. Building on this insight, we propose ReasonCache, a novel KV cache management approach designed to improve the QoS of AI inference systems. ReasonCache utilizes a Collaborative Filtering Algorithm to efficiently identify reusable KV cache blocks and enables zero-copy cache reuse. Experimental evaluation demonstrates that ReasonCache achieves a peak throughput improvement of 89.2% and an average gain of 40-60%, leading to more responsive and cost-effective AI inference services. Notably, this performance is achieved while maintaining higher accuracy compared to existing KV cache management techniques.