A Unified Knowledge-Distillation and Semi-Supervised Learning Framework to Improve Industrial Ads Delivery Systems

TL;DR

This paper introduces UKDSL, a unified framework combining knowledge distillation and semi-supervised learning to enhance industrial ads ranking systems by leveraging larger datasets, reducing overfitting, and improving performance at a massive scale.

Contribution

The paper presents a novel unified framework that integrates knowledge distillation and semi-supervised learning for industrial ads ranking, enabling training on larger datasets and improving efficiency and effectiveness.

Findings

Empirical evidence shows UKDSL reduces prediction bias and miscalibration.

UKDSL enables models to learn from significantly more unlabeled data.

Successful deployment at multi-billion scale demonstrates practical effectiveness.

Abstract

Industrial ads ranking systems conventionally rely on labeled impression data, which leads to challenges such as overfitting, slower incremental gain from model scaling, and biases due to discrepancies between training and serving data. To overcome these issues, we propose a Unified framework for Knowledge-Distillation and Semi-supervised Learning (UKDSL) for ads ranking, empowering the training of models on a significantly larger and more diverse datasets, thereby reducing overfitting and mitigating training-serving data discrepancies. We provide detailed formal analysis and numerical simulations on the inherent miscalibration and prediction bias of multi-stage ranking systems, and show empirical evidence of the proposed framework's capability to mitigate those. Compared to prior work, UKDSL can enable models to learn from a much larger set of unlabeled data, hence, improving the performance while being computationally efficient. Finally, we report the successful deployment of UKDSL in an industrial setting across various ranking models, serving users at multi-billion scale, across various surfaces, geological locations, clients, and optimize for various events, which to the best of our knowledge is the first of its kind in terms of the scale and efficiency at which it operates.