BehaveGPT: A Foundation Model for Large-scale User Behavior Modeling

TL;DR

BehaveGPT is a transformer-based foundational model tailored for large-scale user behavior prediction, leveraging a novel pretraining paradigm to improve generalization and outperform existing methods on real-world datasets.

Contribution

The paper introduces BehaveGPT, a new model with a DRO-based pretraining paradigm specifically designed for user behavior modeling, enabling better capture of complex patterns and transferability.

Findings

Achieves over 10% improvement in macro and weighted recall over baselines.

Demonstrates effective modeling of complex user behavior patterns.

Provides the first scaling law analysis for user behavior models.

Abstract

In recent years, foundational models have revolutionized the fields of language and vision, demonstrating remarkable abilities in understanding and generating complex data; however, similar advances in user behavior modeling have been limited, largely due to the complexity of behavioral data and the challenges involved in capturing intricate temporal and contextual relationships in user activities. To address this, we propose BehaveGPT, a foundational model designed specifically for large-scale user behavior prediction. Leveraging transformer-based architecture and a novel pretraining paradigm, BehaveGPT is trained on vast user behavior datasets, allowing it to learn complex behavior patterns and support a range of downstream tasks, including next behavior prediction, long-term generation, and cross-domain adaptation. Our approach introduces the DRO-based pretraining paradigm tailored for user behavior data, which improves model generalization and transferability by equitably modeling both head and tail behaviors. Extensive experiments on real-world datasets demonstrate that BehaveGPT outperforms state-of-the-art baselines, achieving more than a 10% improvement in macro and weighted recall, showcasing its ability to effectively capture and predict user behavior. Furthermore, we measure the scaling law in the user behavior domain for the first time on the Honor dataset, providing insights into how model performance scales with increased data and parameter sizes.