xGR: Efficient Generative Recommendation Serving at Scale

TL;DR

xGR is a specialized system designed to efficiently serve generative recommendation models at scale, significantly reducing latency and increasing throughput by optimizing the decoding process and parallelism.

Contribution

The paper introduces xGR, a novel recommendation serving system tailored for generative models, with techniques for unified processing, early sorting, and parallelism to meet low-latency demands.

Findings

xGR achieves at least 3.49x throughput over baseline.

xGR maintains low latency under high concurrency.

The system effectively handles long prompt processing and large item spaces.

Abstract

Recommendation system delivers substantial economic benefits by providing personalized predictions. Generative recommendation (GR) integrates LLMs to enhance the understanding of long user-item sequences. Despite employing attention-based architectures, GR's workload differs markedly from that of LLM serving. GR typically processes long prompt while producing short, fixed-length outputs, yet the computational cost of each decode phase is especially high due to the large beam width. In addition, since the beam search involves a vast item space, the sorting overhead becomes particularly time-consuming. We propose xGR, a GR-oriented serving system that meets strict low-latency requirements under highconcurrency scenarios. First, xGR unifies the processing of prefill and decode phases through staged computation and separated KV cache. Second, xGR enables early sorting termination and mask-based item filtering with data structure reuse. Third, xGR reconstructs the overall pipeline to exploit multilevel overlap and multi-stream parallelism. Our experiments with real-world recommendation service datasets demonstrate that xGR achieves at least 3.49x throughput compared to the state-of-the-art baseline under strict latency constraints.