UserBERT: Modeling Long- and Short-Term User Preferences via Self-Supervision

TL;DR

UserBERT is a self-supervised model that captures both long- and short-term user preferences from e-commerce data, improving performance on various downstream tasks without requiring labeled data.

Contribution

The paper introduces a novel extension of BERT for user behavior modeling, integrating long- and short-term preferences in a unified self-supervised framework.

Findings

Significant improvements in downstream task performance

Effective modeling of diverse user behavior sequences

Elimination of labeled data requirement for pre-training

Abstract

E-commerce platforms generate vast amounts of customer behavior data, such as clicks and purchases, from millions of unique users every day. However, effectively using this data for behavior understanding tasks is challenging because there are usually not enough labels to learn from all users in a supervised manner. This paper extends the BERT model to e-commerce user data for pre-training representations in a self-supervised manner. By viewing user actions in sequences as analogous to words in sentences, we extend the existing BERT model to user behavior data. Further, our model adopts a unified structure to simultaneously learn from long-term and short-term user behavior, as well as user attributes. We propose methods for the tokenization of different types of user behavior sequences, the generation of input representation vectors, and a novel pretext task to enable the pre-trained model to learn from its own input, eliminating the need for labeled training data. Extensive experiments demonstrate that the learned representations result in significant improvements when transferred to three different real-world tasks, particularly compared to task-specific modeling and multi-task representation learning