Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm

TL;DR

This paper introduces an unbiased pairwise learning-to-rank algorithm that jointly estimates position biases in click data and trains a high-performance ranker, improving search relevance and reducing bias effects.

Contribution

It presents the first pairwise learning-to-rank method that simultaneously debiases click data and trains a ranker, outperforming existing algorithms in experiments.

Findings

Significantly outperforms existing unbiased learning-to-rank algorithms on benchmark data.

Effectively debiases click data in online search engine experiments.

Enhances relevance ranking by reducing position bias in click data.

Abstract

Although click data is widely used in search systems in practice, so far the inherent bias, most notably position bias, has prevented it from being used in training of a ranker for search, i.e., learning-to-rank. Recently, a number of authors have proposed new techniques referred to as 'unbiased learning-to-rank', which can reduce position bias and train a relatively high-performance ranker using click data. Most of the algorithms, based on the inverse propensity weighting (IPW) principle, first estimate the click bias at each position, and then train an unbiased ranker with the estimated biases using a learning-to-rank algorithm. However, there has not been a method for pairwise learning-to-rank that can jointly conduct debiasing of click data and training of a ranker using a pairwise loss function. In this paper, we propose a novel algorithm, which can jointly estimate the biases at click positions and the biases at unclick positions, and learn an unbiased ranker. Experiments on benchmark data show that our algorithm can significantly outperform existing algorithms. In addition, an online A/B Testing at a commercial search engine shows that our algorithm can effectively conduct debiasing of click data and enhance relevance ranking.