RcLLM: Accelerating Generative Recommendation via Beyond-Prefix KV Caching

TL;DR

RcLLM introduces a distributed inference system that accelerates generative recommendation by decomposing prompts into reusable blocks and optimizing caching and attention mechanisms, significantly reducing latency.

Contribution

The paper proposes RcLLM, a novel system that enhances generative recommendation efficiency through Beyond-Prefix KV Caching and stratified distributed storage, enabling real-time deployment.

Findings

Reduces Time-To-First-Token (TTFT) by up to 9.51x.

Supports large item catalogs with a stratified distributed storage design.

Maintains recommendation accuracy while significantly improving latency.

Abstract

Large Language Models (LLMs) are transforming recommendation from ranking into a generative task, but industrial deployment remains limited by the high latency of processing long, personalized prompts. Standard prefix caching provides limited benefit because reuse in recommendation workloads is often non-contiguous across user histories and item contexts. We present RcLLM, a distributed inference system for generative recommendation with Beyond-Prefix KV Caching. RcLLM decomposes prompts into reusable blocks and supports large item catalogs with a stratified distributed storage design: compact user-history caches are replicated for zero-latency retrieval, while massive item caches are sharded using similarity-aware placement. To reduce redundant quadratic attention computation, RcLLM combines an affinity-based global scheduler that improves data locality with a selective attention mechanism that corrects approximation errors. Experiments on real-world datasets show that RcLLM reduces Time-To-First-Token (TTFT) by 1.31x-9.51x compared with state-of-the-art prefix caching systems, enabling real-time serving with negligible impact on recommendation accuracy.