MoE-MLoRA for Multi-Domain CTR Prediction: Efficient Adaptation with Expert Specialization

TL;DR

MoE-MLoRA introduces a mixture-of-experts framework for multi-domain CTR prediction, enabling domain-specific expert specialization and dynamic gating to improve performance in large-scale, diverse datasets.

Contribution

It presents a novel expert-based adaptation method that enhances multi-domain recommendation models through independent expert training and adaptive gating.

Findings

Improves Weighed-AUC by +1.45 in Taobao-20 dataset.

Larger ensembles do not always yield better performance.

Effectiveness varies with dataset domain diversity and sparsity.

Abstract

Personalized recommendation systems must adapt to user interactions across different domains. Traditional approaches like MLoRA apply a single adaptation per domain but lack flexibility in handling diverse user behaviors. To address this, we propose MoE-MLoRA, a mixture-of-experts framework where each expert is first trained independently to specialize in its domain before a gating network is trained to weight their contributions dynamically. We evaluate MoE-MLoRA across eight CTR models on Movielens and Taobao, showing that it improves performance in large-scale, dynamic datasets (+1.45 Weighed-AUC in Taobao-20) but offers limited benefits in structured datasets with low domain diversity and sparsity. Further analysis of the number of experts per domain reveals that larger ensembles do not always improve performance, indicating the need for model-aware tuning. Our findings highlight the potential of expert-based architectures for multi-domain recommendation systems, demonstrating that task-aware specialization and adaptive gating can enhance predictive accuracy in complex environments. The implementation and code are available in our GitHub repository.