LUMOS: Large User MOdels for User Behavior Prediction

TL;DR

LUMOS is a transformer-based model that predicts user behavior at scale by jointly learning multiple tasks from raw activity data, using novel mechanisms to incorporate future events and multi-modal information, leading to improved accuracy and business impact.

Contribution

The paper introduces LUMOS, a scalable, multi-task transformer architecture that eliminates manual feature engineering and task-specific models for user behavior prediction.

Findings

LUMOS outperforms traditional models with an average ROC-AUC improvement of 0.025.

Achieves a 4.6% reduction in MAPE across tasks.

Online A/B testing shows a 3.15% increase in Daily Active Users.

Abstract

User behavior prediction at scale remains a critical challenge for online B2C platforms. Traditional approaches rely heavily on task-specific models and domain-specific feature engineering. This is time-consuming, computationally expensive, and requires domain expertise and therefore, not scalable. We present LUMOS (Large User MOdel Series), a transformer-based architecture that eliminates task-specific models and manual feature engineering by learning multiple tasks jointly using only raw user activity data. LUMOS introduces a novel cross-attention mechanism that conditions predictions on future known events (e.g., holidays, sales, etc.), enabling the model to predict complex behavior patterns like "how will upcoming holidays affect user engagement?" The architecture also employs multi-modal tokenization, combining user activities, event context, and static user demographic attributes into rich representations processed through specialized embedding pathways. Through extensive experiments on a production dataset spanning 1.7 trillion user activity tokens from 250 million users, we demonstrate that LUMOS achieves superior performance compared to traditional task-specific models. Across 5 tasks with established baselines, we achieve an average improvement of 0.025 in ROC-AUC for binary classification tasks and 4.6\% reduction in MAPE for regression tasks. Online A/B testing validates these improvements translate to measurable business impact with a 3.15\% increase in Daily Active Users.