Beyond Self-Consistency: Loss-Balanced Perturbation-Based Regularization Improves Industrial-Scale Ads Ranking

TL;DR

This paper introduces a novel perturbation-based regularization method called Loss-Balanced Small Perturbation Regularization (LSPR) for large-scale ads ranking models, demonstrating its scalability and superior performance over existing self-consistency approaches in industrial settings.

Contribution

The paper presents the first successful application of perturbation-based regularization in large-scale ads ranking and proposes a new scalable algorithm, LSPR, that outperforms existing methods.

Findings

LSPR outperforms SCR across various experimental setups.

Both SCR and LSPR are scalable to industrial-scale systems.

LSPR is successfully applied in a billion-scale ranking system.

Abstract

Perturbation-based regularization techniques address many challenges in industrial-scale large models, particularly with sparse labels, and emphasize consistency and invariance for perturbation in model predictions. One of the popular regularization techniques has been various forms of self-consistency, which involve making small modifications to input data while preserving contextual information and enforcing similar predictions through auxiliary loss functions. In this work, we explore the first successful application of perturbation-based regularization algorithms in large-scale ads ranking models, and further propose a novel regularization algorithm, namely, Loss-Balanced Small Perturbation Regularization (LSPR) that can be used in potentially any deep learning model. We have successfully demonstrate that both Self-Consistency Regularization approaches (SCR) and LSPR are scalable and can improve ads delivery systems. By conducting industrial-scale experiments, and numerical analysis, we additionally show that our proposed LSPR, performs consistently better compared to SCR, across various groups and signal availability setups. Finally, we report a successful application of the proposed LSPR in a billion-scale industrial ranking system, which to the best of our knowledge, is the first of its kind, and it is specially designed to address the various scalability challenges (e.g, various surfaces, geological locations, clients and so on) as we will mention in this paper.