Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction

TL;DR

This paper introduces Propensity Ratio Scoring (PRS), a new method that corrects bias in learning to rank by addressing both clicks and non-clicks, leading to more effective and unbiased ranker training.

Contribution

The paper rigorously proves the bias caused by ignoring non-clicks and proposes PRS, a novel weighting scheme that improves unbiased learning to rank by treating both clicks and non-clicks.

Findings

PRS outperforms existing methods on synthetic LTR benchmarks.

PRS achieves better real-world ranking performance in GMail search.

The method reduces unnecessary relevant-relevant document comparisons.

Abstract

Recent advances in unbiased learning to rank (LTR) count on Inverse Propensity Scoring (IPS) to eliminate bias in implicit feedback. Though theoretically sound in correcting the bias introduced by treating clicked documents as relevant, IPS ignores the bias caused by (implicitly) treating non-clicked ones as irrelevant. In this work, we first rigorously prove that such use of click data leads to unnecessary pairwise comparisons between relevant documents, which prevent unbiased ranker optimization. Based on the proof, we derive a simple yet well justified new weighting scheme, called Propensity Ratio Scoring (PRS), which provides treatments on both clicks and non-clicks. Besides correcting the bias in clicks, PRS avoids relevant-relevant document comparisons in LTR training and enjoys a lower variability. Our extensive empirical evaluations confirm that PRS ensures a more effective use of click data and improved performance in both synthetic data from a set of LTR benchmarks, as well as in the real-world large-scale data from GMail search.