A General Framework for Pairwise Unbiased Learning to Rank

TL;DR

This paper introduces a flexible, assumption-minimal framework for pairwise unbiased learning to rank, broadening applicability across diverse user behaviors and layouts, and demonstrating improved robustness over existing methods.

Contribution

It develops a simplified, more broadly applicable version of Unbiased LambdaMART that maintains unbiasedness under fewer assumptions about user examination patterns.

Findings

The new framework applies to a wider range of user browsing behaviors.

The simplified algorithm retains unbiasedness in diverse settings.

It outperforms the original Unbiased LambdaMART when examination independence is not assumed.

Abstract

Pairwise debiasing is one of the most effective strategies in reducing position bias in learning-to-rank (LTR) models. However, limiting the scope of this strategy, are the underlying assumptions required by many pairwise debiasing approaches. In this paper, we develop an approach based on a minimalistic set of assumptions that can be applied to a much broader range of user browsing patterns and arbitrary presentation layouts. We implement the approach as a simplified version of the Unbiased LambdaMART and demonstrate that it retains the underlying unbiasedness property in a wider variety of settings than the original algorithm. Finally, using simulations with "golden" relevance labels, we will show that the simplified version compares favourably with the original Unbiased LambdaMART when the examination of different positions in a ranked list is not assumed to be independent.