MDL: A Unified Multi-Distribution Learner in Large-scale Industrial Recommendation through Tokenization

TL;DR

This paper introduces MDL, a unified multi-distribution learning framework for industrial recommendation systems that leverages tokenization and attention mechanisms to improve multi-scenario and multi-task modeling, achieving significant performance gains.

Contribution

The paper proposes a novel tokenization-based approach inspired by LLM prompting, enabling deep interaction between features, scenarios, and tasks within a unified model for industrial recommendation.

Findings

MDL outperforms state-of-the-art baselines on real-world datasets.

Online A/B tests show improved user engagement and reduced query change rate.

MDL is deployed in production, serving hundreds of millions of users.

Abstract

Industrial recommender systems increasingly adopt multi-scenario learning (MSL) and multi-task learning (MTL) to handle diverse user interactions and contexts, but existing approaches suffer from two critical drawbacks: (1) underutilization of large-scale model parameters due to limited interaction with complex feature modules, and (2) difficulty in jointly modeling scenario and task information in a unified framework. To address these challenges, we propose a unified \textbf{M}ulti-\textbf{D}istribution \textbf{L}earning (MDL) framework, inspired by the "prompting" paradigm in large language models (LLMs). MDL treats scenario and task information as specialized tokens rather than auxiliary inputs or gating signals. Specifically, we introduce a unified information tokenization module that transforms features, scenarios, and tasks into a unified tokenized format. To facilitate deep interaction, we design three synergistic mechanisms: (1) feature token self-attention for rich feature interactions, (2) domain-feature attention for scenario/task-adaptive feature activation, and (3) domain-fused aggregation for joint distribution prediction. By stacking these interactions, MDL enables scenario and task information to "prompt" and activate the model's vast parameter space in a bottom-up, layer-wise manner. Extensive experiments on real-world industrial datasets demonstrate that MDL significantly outperforms state-of-the-art MSL and MTL baselines. Online A/B testing on Douyin Search platform over one month yields +0.0626\% improvement in LT30 and -0.3267\% reduction in change query rate. MDL has been fully deployed in production, serving hundreds of millions of users daily.