TurboGR: An Accelerated Training System for Large-Scale Generative Recommendation

TL;DR

TurboGR introduces an optimized training system for large-scale generative recommendation on Ascend NPUs, overcoming system bottlenecks with innovative acceleration, communication, and negative sampling techniques.

Contribution

The paper presents extit{TurboGR}, a system that systematically addresses Ascend NPU challenges for scalable generative recommendation training with three core innovations.

Findings

Supports training up to 0.2B parameters with high efficiency.

Achieves 54.71% MFU and near-linear scalability (0.97).

Reduces inter-device imbalance from 47% to 2.4%.

Abstract

Generative recommendation (GR) has emerged as a promising paradigm that replaces fragmented, scenario-specific architectures with unified Transformer-based models, exhibiting scaling-law behavior where recommendation quality improves systematically with increased model capacity and training data. However, deploying GR at scale on Ascend NPUs faces fundamental system-level challenges. These challenges are further exacerbated on Ascend NPUs due to the absence of high-performance implementations for jagged operators and the architectural mismatch between irregular sparse primitives and NPU's dense-computation-optimized design. In this paper, we present \model, an Ascend-affinity training system for generative recommendation that systematically addresses these bottlenecks through three core innovations: (i) Ascend-affinity jagged acceleration, including fusion operators that eliminate padding redundancy and dynamic load balancing that reduces inter-device imbalance from 47\% to 2.4\%; (ii) distributed communication optimization, comprising hierarchical sparse parallelism, semi-asynchronous training with proven convergence guarantees, and fine-grained pipeline orchestration that sustains 94\% NPU utilization; and (iii) negative sampling optimization via asynchronous offloading, jaggedness-aware FP16 quantization, and intra-batch logit sharing that expand the effective negative space without additional embedding lookups. Evaluated on the KuaiRand-27K dataset, \model supports training at up to 0.2B parameters and achieves 54.71\% MFU with near-linear scalability (0.97).