Genie: An Open Box Counterfactual Policy Estimator for Optimizing Sponsored Search Marketplace

TL;DR

Genie is an offline counterfactual policy estimation framework that uses an open box simulation engine to optimize sponsored search marketplaces, outperforming existing methods and enabling safe policy tuning without risky experiments.

Contribution

Introducing Genie, a novel open box simulation-based framework for offline policy estimation that improves optimization efficiency and safety in sponsored search marketplaces.

Findings

Genie outperforms observational approaches on Bing traffic.

It enables safe tuning of new policies without risky experiments.

It handles frequent policy updates effectively.

Abstract

In this paper, we propose an offline counterfactual policy estimation framework called Genie to optimize Sponsored Search Marketplace. Genie employs an open box simulation engine with click calibration model to compute the KPI impact of any modification to the system. From the experimental results on Bing traffic, we showed that Genie performs better than existing observational approaches that employs randomized experiments for traffic slices that have frequent policy updates. We also show that Genie can be used to tune completely new policies efficiently without creating risky randomized experiments due to cold start problem. As time of today, Genie hosts more than 10000 optimization jobs yearly which runs more than 30 Million processing node hours of big data jobs for Bing Ads. For the last 3 years, Genie has been proven to be the one of the major platforms to optimize Bing Ads Marketplace due to its reliability under frequent policy changes and its efficiency to minimize risks in real experiments.