SwiftPruner: Reinforced Evolutionary Pruning for Efficient Ad Relevance

TL;DR

This paper introduces SwiftPruner, a reinforcement learning-based framework for structured pruning of BERT models, enabling efficient real-time ad relevance inference for cold start ads with improved accuracy and latency.

Contribution

It proposes a novel reinforcement learning approach with a latency-aware reward to optimize layer-wise sparsity in BERT for low-latency ad relevance tasks.

Findings

Achieves higher ROC AUC than uniform sparsity baselines.

Reduces cold start ad defect ratio by 11.7%.

Meets latency constraints with improved model performance.

Abstract

Ad relevance modeling plays a critical role in online advertising systems including Microsoft Bing. To leverage powerful transformers like BERT in this low-latency setting, many existing approaches perform ad-side computations offline. While efficient, these approaches are unable to serve cold start ads, resulting in poor relevance predictions for such ads. This work aims to design a new, low-latency BERT via structured pruning to empower real-time online inference for cold start ads relevance on a CPU platform. Our challenge is that previous methods typically prune all layers of the transformer to a high, uniform sparsity, thereby producing models which cannot achieve satisfactory inference speed with an acceptable accuracy. In this paper, we propose SwiftPruner - an efficient framework that leverages evolution-based search to automatically find the best-performing layer-wise sparse BERT model under the desired latency constraint. Different from existing evolution algorithms that conduct random mutations, we propose a reinforced mutator with a latency-aware multi-objective reward to conduct better mutations for efficiently searching the large space of layer-wise sparse models. Extensive experiments demonstrate that our method consistently achieves higher ROC AUC and lower latency than the uniform sparse baseline and state-of-the-art search methods. Remarkably, under our latency requirement of 1900us on CPU, SwiftPruner achieves a 0.86% higher AUC than the state-of-the-art uniform sparse baseline for BERT-Mini on a large scale real-world dataset. Online A/B testing shows that our model also achieves a significant 11.7% cut in the ratio of defective cold start ads with satisfactory real-time serving latency.