Consistent Position Bias Estimation without Online Interventions for Learning-to-Rank

TL;DR

This paper introduces a novel method for accurately estimating position bias in learning-to-rank systems using implicit feedback data from multiple ranking functions, eliminating the need for manual judgments or interventions.

Contribution

It presents the first consistent propensity estimation technique that works without manual relevance judgments, interventions, or restrictive assumptions, applicable to context-aware models.

Findings

Method is scalable and robust in simulations

Achieves accurate propensity estimates without manual relevance data

Applicable to extended context-dependent propensity models

Abstract

Presentation bias is one of the key challenges when learning from implicit feedback in search engines, as it confounds the relevance signal with uninformative signals due to position in the ranking, saliency, and other presentation factors. While it was recently shown how counterfactual learning-to-rank (LTR) approaches \cite{Joachims/etal/17a} can provably overcome presentation bias if observation propensities are known, it remains to show how to accurately estimate these propensities. In this paper, we propose the first method for producing consistent propensity estimates without manual relevance judgments, disruptive interventions, or restrictive relevance modeling assumptions. We merely require that we have implicit feedback data from multiple different ranking functions. Furthermore, we argue that our estimation technique applies to an extended class of Contextual Position-Based Propensity Models, where propensities not only depend on position but also on observable features of the query and document. Initial simulation studies confirm that the approach is scalable, accurate, and robust.