Counterfactual Evaluation of Ads Ranking Models through Domain Adaptation

TL;DR

This paper introduces a domain-adapted reward model for offline evaluation of ads ranking models, enabling effective measurement of model changes in large-scale recommender systems where traditional methods are impractical.

Contribution

The paper presents a novel domain-adapted reward model that enhances offline evaluation accuracy for ranking models in large-scale ad systems, outperforming existing IPS and non-generalized reward approaches.

Findings

The proposed method outperforms vanilla IPS in large-scale settings.

It provides more accurate reward estimation for ranking model changes.

The approach is effective in real-world Ads recommender systems.

Abstract

We propose a domain-adapted reward model that works alongside an Offline A/B testing system for evaluating ranking models. This approach effectively measures reward for ranking model changes in large-scale Ads recommender systems, where model-free methods like IPS are not feasible. Our experiments demonstrate that the proposed technique outperforms both the vanilla IPS method and approaches using non-generalized reward models.