Analysis of Multivariate Scoring Functions for Automatic Unbiased Learning to Rank

TL;DR

This paper investigates the use of multivariate scoring functions in automatic unbiased learning to rank (AutoULTR), demonstrating that permutation-invariant multivariate models outperform univariate and permutation-variant models in noisy click data scenarios.

Contribution

It provides a theoretical analysis of multivariate scoring functions in AutoULTR, highlighting permutation invariance as a key factor for effectiveness.

Findings

Permutation-invariant multivariate models outperform univariate models.

Permutation-variant multivariate models perform worse than permutation-invariant ones.

AutoULTR with permutation-invariant multivariate functions significantly improves ranking accuracy.

Abstract

Leveraging biased click data for optimizing learning to rank systems has been a popular approach in information retrieval. Because click data is often noisy and biased, a variety of methods have been proposed to construct unbiased learning to rank (ULTR) algorithms for the learning of unbiased ranking models. Among them, automatic unbiased learning to rank (AutoULTR) algorithms that jointly learn user bias models (i.e., propensity models) with unbiased rankers have received a lot of attention due to their superior performance and low deployment cost in practice. Despite their differences in theories and algorithm design, existing studies on ULTR usually use uni-variate ranking functions to score each document or result independently. On the other hand, recent advances in context-aware learning-to-rank models have shown that multivariate scoring functions, which read multiple documents together and predict their ranking scores jointly, are more powerful than uni-variate ranking functions in ranking tasks with human-annotated relevance labels. Whether such superior performance would hold in ULTR with noisy data, however, is mostly unknown. In this paper, we investigate existing multivariate scoring functions and AutoULTR algorithms in theory and prove that permutation invariance is a crucial factor that determines whether a context-aware learning-to-rank model could be applied to existing AutoULTR framework. Our experiments with synthetic clicks on two large-scale benchmark datasets show that AutoULTR models with permutation-invariant multivariate scoring functions significantly outperform those with uni-variate scoring functions and permutation-variant multivariate scoring functions.