SML:Enhance the Network Smoothness with Skip Meta Logit for CTR Prediction

TL;DR

This paper introduces Skip Meta Logit, a novel method that enhances network smoothness for CTR prediction by incorporating skip connections and normalization, leading to improved accuracy and stability in real-world applications.

Contribution

The paper proposes Skip Meta Logit with Meta Tanh Normalization, providing a new way to improve deep neural network training stability and performance for CTR prediction.

Findings

Achieved incremental performance boosts on state-of-the-art models.

Proved the absence of spurious local optima in deep skip logit networks.

Delivered offline and online metrics improvements in TikTok's ad system.

Abstract

In light of the smoothness property brought by skip connections in ResNet, this paper proposed the Skip Logit to introduce the skip connection mechanism that fits arbitrary DNN dimensions and embraces similar properties to ResNet. Meta Tanh Normalization (MTN) is designed to learn variance information and stabilize the training process. With these delicate designs, our Skip Meta Logit (SML) brought incremental boosts to the performance of extensive SOTA ctr prediction models on two real-world datasets. In the meantime, we prove that the optimization landscape of arbitrarily deep skip logit networks has no spurious local optima. Finally, SML can be easily added to building blocks and has delivered offline accuracy and online business metrics gains on app ads learning to rank systems at TikTok.