Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data

TL;DR

This paper introduces a novel pretraining strategy for user behavior sequence modeling that automatically constructs supervision embeddings, eliminating manual vocabulary selection and improving performance and efficiency in industrial applications.

Contribution

The paper proposes Bootstrapping Your Behavior, a new pretraining method that automatically generates supervision embeddings using a student-teacher encoder scheme, removing the need for manual behavior vocabulary construction.

Findings

Achieves 3.9% average AUC improvement on real-world datasets.

Increases training throughput by 98.9%.

Enhances online risk prediction accuracy, reducing bad debt risk.

Abstract

User Behavior Sequence (UBS) modeling is crucial in industrial applications. As data scale and task diversity grow, UBS pretraining methods have become increasingly pivotal. State-of-the-art UBS pretraining methods rely on predicting behavior distributions. The key step in these methods is constructing a selected behavior vocabulary. However, this manual step is labor-intensive and prone to bias. The limitation of vocabulary capacity also directly affects models' generalization ability. In this paper, we introduce Bootstrapping Your Behavior (\model{}), a novel UBS pretraining strategy that predicts an automatically constructed supervision embedding summarizing all behaviors' information within a future time window, eliminating the manual behavior vocabulary selection. In implementation, we incorporate a student-teacher encoder scheme to construct the pretraining supervision effectively. Experiments on two real-world industrial datasets and eight downstream tasks demonstrate that \model{} achieves an average improvement of 3.9\% in AUC and 98.9\% in training throughput. Notably, the model exhibits meaningful attention patterns and cluster representations during pretraining without any label supervision. In our online deployment over two months, the pretrained model improves the KS by about 2.7\% and 7.1\% over the baseline model for two financial overdue risk prediction tasks in the Alipay mobile application, which reduces bad debt risk by millions of dollars for Ant group.