AutoADR: Automatic Model Design for Ad Relevance

TL;DR

AutoADR is an end-to-end framework that uses neural architecture search guided by knowledge distillation to design efficient, high-performing models for online Ad Relevance, successfully deployed in Microsoft Bing with significant improvements.

Contribution

It introduces a novel AutoML-based approach integrating neural architecture search with knowledge distillation for Ad Relevance model design, optimized for online constraints.

Findings

Improved PR AUC by 2.65x over baseline.

Reduced Bad-Ad ratio by 4.6% in online A/B testing.

Successfully deployed in Microsoft Bing Ad Relevance system.

Abstract

Large-scale pre-trained models have attracted extensive attention in the research community and shown promising results on various tasks of natural language processing. However, these pre-trained models are memory and computation intensive, hindering their deployment into industrial online systems like Ad Relevance. Meanwhile, how to design an effective yet efficient model architecture is another challenging problem in online Ad Relevance. Recently, AutoML shed new lights on architecture design, but how to integrate it with pre-trained language models remains unsettled. In this paper, we propose AutoADR (Automatic model design for AD Relevance) -- a novel end-to-end framework to address this challenge, and share our experience to ship these cutting-edge techniques into online Ad Relevance system at Microsoft Bing. Specifically, AutoADR leverages a one-shot neural architecture search algorithm to find a tailored network architecture for Ad Relevance. The search process is simultaneously guided by knowledge distillation from a large pre-trained teacher model (e.g. BERT), while taking the online serving constraints (e.g. memory and latency) into consideration. We add the model designed by AutoADR as a sub-model into the production Ad Relevance model. This additional sub-model improves the Precision-Recall AUC (PR AUC) on top of the original Ad Relevance model by 2.65X of the normalized shipping bar. More importantly, adding this automatically designed sub-model leads to a statistically significant 4.6% Bad-Ad ratio reduction in online A/B testing. This model has been shipped into Microsoft Bing Ad Relevance Production model.