Abacus: Self-Supervised Event Counting-Aligned Distributional Pretraining for Sequential User Modeling

TL;DR

This paper introduces Abacus, a self-supervised pretraining method that predicts user event frequency distributions to improve sequential user modeling in display advertising, leading to better performance and faster convergence.

Contribution

The paper proposes Abacus, a novel self-supervised pretraining approach that models user event distributions, enhancing sequential models for advertising with improved accuracy and efficiency.

Findings

Abacus pretraining outperforms existing methods in AUC.

Hybrid objective improves downstream task convergence.

Up to +6.1% AUC gain over baselines.

Abstract

Modeling user purchase behavior is a critical challenge in display advertising systems, necessary for real-time bidding. The difficulty arises from the sparsity of positive user events and the stochasticity of user actions, leading to severe class imbalance and irregular event timing. Predictive systems usually rely on hand-crafted "counter" features, overlooking the fine-grained temporal evolution of user intent. Meanwhile, current sequential models extract direct sequential signal, missing useful event-counting statistics. We enhance deep sequential models with self-supervised pretraining strategies for display advertising. Especially, we introduce Abacus, a novel approach of predicting the empirical frequency distribution of user events. We further propose a hybrid objective unifying Abacus with sequential learning objectives, combining stability of aggregated statistics with the sequence modeling sensitivity. Experiments on two real-world datasets show that Abacus pretraining outperforms existing methods accelerating downstream task convergence, while hybrid approach yields up to +6.1% AUC compared to the baselines.