Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models

TL;DR

This paper investigates the one-epoch overfitting phenomenon in deep CTR prediction models, revealing factors like model structure, optimization algorithm, and feature sparsity that influence this behavior, with implications for training strategies.

Contribution

It identifies and analyzes the one-epoch overfitting issue in CTR models, providing experimental insights and hypotheses to guide future training improvements.

Findings

One-epoch overfitting causes performance degradation in CTR models.

Model structure and optimization algorithm significantly affect overfitting.

Feature sparsity is closely related to the overfitting phenomenon.

Abstract

Deep learning techniques have been applied widely in industrial recommendation systems. However, far less attention has been paid to the overfitting problem of models in recommendation systems, which, on the contrary, is recognized as a critical issue for deep neural networks. In the context of Click-Through Rate (CTR) prediction, we observe an interesting one-epoch overfitting problem: the model performance exhibits a dramatic degradation at the beginning of the second epoch. Such a phenomenon has been witnessed widely in real-world applications of CTR models. Thereby, the best performance is usually achieved by training with only one epoch. To understand the underlying factors behind the one-epoch phenomenon, we conduct extensive experiments on the production data set collected from the display advertising system of Alibaba. The results show that the model structure, the optimization algorithm with a fast convergence rate, and the feature sparsity are closely related to the one-epoch phenomenon. We also provide a likely hypothesis for explaining such a phenomenon and conduct a set of proof-of-concept experiments. We hope this work can shed light on future research on training more epochs for better performance.