Towards Disentangling Relevance and Bias in Unbiased Learning to Rank

TL;DR

This paper addresses the challenge of disentangling relevance and bias in unbiased learning to rank, revealing confounding issues in existing models and proposing methods to improve bias mitigation.

Contribution

It identifies a critical confounding problem in two-tower ULTR models and introduces three novel methods to better disentangle relevance from bias.

Findings

The bias tower can be confounded with relevance due to true relevance signals.

The proposed methods effectively mitigate confounding effects.

Empirical results show improved relevance estimation and bias reduction.

Abstract

Unbiased learning to rank (ULTR) studies the problem of mitigating various biases from implicit user feedback data such as clicks, and has been receiving considerable attention recently. A popular ULTR approach for real-world applications uses a two-tower architecture, where click modeling is factorized into a relevance tower with regular input features, and a bias tower with bias-relevant inputs such as the position of a document. A successful factorization will allow the relevance tower to be exempt from biases. In this work, we identify a critical issue that existing ULTR methods ignored - the bias tower can be confounded with the relevance tower via the underlying true relevance. In particular, the positions were determined by the logging policy, i.e., the previous production model, which would possess relevance information. We give both theoretical analysis and empirical results to show the negative effects on relevance tower due to such a correlation. We then propose three methods to mitigate the negative confounding effects by better disentangling relevance and bias. Empirical results on both controlled public datasets and a large-scale industry dataset show the effectiveness of the proposed approaches.