Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction

TL;DR

This paper systematically explores the impact of various normalization techniques on deep neural networks for click-through rate prediction, demonstrating that proper normalization significantly improves performance and introducing a novel normalization method called VO-LN.

Contribution

The paper provides a comprehensive analysis of normalization effects in DNNs for CTR, proposes a new normalization method VO-LN, and introduces the NormDNN model that leverages normalization for enhanced accuracy.

Findings

Proper normalization significantly improves DNN performance in CTR tasks.

Variance in normalization is the key factor influencing model effectiveness.

The proposed VO-LN normalization outperforms traditional methods.

Abstract

Normalization has become one of the most fundamental components in many deep neural networks for machine learning tasks while deep neural network has also been widely used in CTR estimation field. Among most of the proposed deep neural network models, few model utilize normalization approaches. Though some works such as Deep & Cross Network (DCN) and Neural Factorization Machine (NFM) use Batch Normalization in MLP part of the structure, there isn't work to thoroughly explore the effect of the normalization on the DNN ranking systems. In this paper, we conduct a systematic study on the effect of widely used normalization schemas by applying the various normalization approaches to both feature embedding and MLP part in DNN model. Extensive experiments are conduct on three real-world datasets and the experiment results demonstrate that the correct normalization significantly enhances model's performance. We also propose a new and effective normalization approaches based on LayerNorm named variance only LayerNorm(VO-LN) in this work. A normalization enhanced DNN model named NormDNN is also proposed based on the above-mentioned observation. As for the reason why normalization works for DNN models in CTR estimation, we find that the variance of normalization plays the main role and give an explanation in this work.