PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving

TL;DR

PCR is a system that enhances cache reuse in RAG serving by using intelligent prefetching and pipelining, significantly reducing latency and improving throughput in large language model applications.

Contribution

PCR introduces a novel prefix-tree caching, layer-wise overlapping, and queue-based prefetching approach to maximize KV-cache reuse efficiency in RAG systems.

Findings

Achieves up to 2.47x speedup in TTFT.

Outperforms existing KV-cache reuse methods.

Effectively reduces latency in high-throughput RAG serving.

Abstract

Retrieval-Augmented Generation (RAG) systems enhance the performance of large language models (LLMs) by incorporating supplementary retrieved documents, enabling more accurate and context-aware responses. However, integrating these external documents often results in very long input sequences, which significantly increases computation costs during the prefill stage, where key-value (KV) representations for all input tokens are generated. This latency bottleneck becomes especially pronounced under high-throughput serving scenarios. KV-cache reuse offers a promising solution by storing previously computed KV states for shared input prefixes, thereby avoiding redundant computation across requests that contain overlapping context. Yet, the effectiveness of cache reuse is often limited by three practical challenges: low cache hit rates due to naive eviction policies, high CPU-GPU data transfer overhead, and slow SSD I/O when caches spill to storage. To address these issues, we propose PCR, a system designed to maximize KV-cache reuse efficiency through intelligent prefetching and pipelined data movement. Specifically, PCR introduces three key techniques: (1) a prefix-tree caching structure with a look-ahead LRU replacement policy that uses pending requests in the scheduler queue to improve cache hit ratios; (2) layer-wise overlapping that pipelines KV-cache loading and GPU computation across CUDA streams to hide communication latency; and (3) queue-based prefetching that proactively loads relevant KV caches from SSD into DRAM before they are needed. Extensive experiments show that PCR outperforms existing KV-cache reuse methods, achieving up to a 2.47x speedup in terms of average TTFT.