Streaming CTR Prediction: Rethinking Recommendation Task for Real-World Streaming Data

TL;DR

This paper redefines CTR prediction for real-world streaming data, introduces new benchmarks and metrics, and proposes methods to improve model robustness amid challenges like distribution shift and non-stationarity.

Contribution

It formulates streaming CTR prediction as a new task, establishes dedicated benchmarks, and offers simple methods to enhance model performance in dynamic streaming environments.

Findings

Existence of the 'streaming learning dilemma' affecting model performance

Parameter tuning and exemplar replay significantly improve streaming CTR models

Differences in factor effects between static and streaming scenarios

Abstract

The Click-Through Rate (CTR) prediction task is critical in industrial recommender systems, where models are usually deployed on dynamic streaming data in practical applications. Such streaming data in real-world recommender systems face many challenges, such as distribution shift, temporal non-stationarity, and systematic biases, which bring difficulties to the training and utilizing of recommendation models. However, most existing studies approach the CTR prediction as a classification task on static datasets, assuming that the train and test sets are independent and identically distributed (a.k.a, i.i.d. assumption). To bridge this gap, we formulate the CTR prediction problem in streaming scenarios as a Streaming CTR Prediction task. Accordingly, we propose dedicated benchmark settings and metrics to evaluate and analyze the performance of the models in streaming data. To better understand the differences compared to traditional CTR prediction tasks, we delve into the factors that may affect the model performance, such as parameter scale, normalization, regularization, etc. The results reveal the existence of the ''streaming learning dilemma'', whereby the same factor may have different effects on model performance in the static and streaming scenarios. Based on the findings, we propose two simple but inspiring methods (i.e., tuning key parameters and exemplar replay) that significantly improve the effectiveness of the CTR models in the new streaming scenario. We hope our work will inspire further research on streaming CTR prediction and help improve the robustness and adaptability of recommender systems.