TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders

TL;DR

TokenMixer-Large is a new large-scale recommendation model that overcomes previous limitations in scalability and efficiency, achieving significant online and offline performance improvements in industrial settings.

Contribution

It introduces a novel architecture with mixing-and-reverting operations, residuals, auxiliary loss, and sparse MoE for scalable, efficient recommendation modeling.

Findings

Scaled to 7B and 15B parameters with successful deployment

Achieved +1.66% in orders and +2.98% in GMV in e-commerce

Improved advertising and live streaming revenue metrics

Abstract

While scaling laws for recommendation models have gained significant traction, existing architectures such as Wukong, HiFormer and DHEN, often struggle with sub-optimal designs and hardware under-utilization, limiting their practical scalability. Our previous TokenMixer architecture (introduced in RankMixer paper) addressed effectiveness and efficiency by replacing self-attention with a ightweight token-mixing operator; however, it faced critical bottlenecks in deeper configurations, including sub-optimal residual paths, vanishing gradients, incomplete MoE sparsification and constrained scalability. In this paper, we propose TokenMixer-Large, a systematically evolved architecture designed for extreme-scale recommendation. By introducing a mixing-and-reverting operation, inter-layer residuals and the auxiliary loss, we ensure stable gradient propagation even as model depth increases. Furthermore, we incorporate a Sparse Per-token MoE to enable efficient parameter expansion. TokenMixer-Large successfully scales its parameters to 7-billion and 15-billion on online traffic and offline experiments, respectively. Currently deployed in multiple scenarios at ByteDance, TokenMixer-Large has achieved significant offline and online performance gains, delivering an increase of +1.66\% in orders and +2.98\% in per-capita preview payment GMV for e-commerce, improving ADSS by +2.0\% in advertising and achieving a +1.4\% revenue growth for live streaming.