Noise tolerance of learning to rank under class-conditional label noise

TL;DR

This paper introduces noise-tolerant loss functions for learning-to-rank models that remain effective under class-conditional label noise, supported by theoretical analysis and experimental validation.

Contribution

It proposes a class of noise-tolerant LtR loss functions with proven consistency under label noise, extending practical robustness in noisy data scenarios.

Findings

Empirical risk minimization remains consistent with the proposed losses under label noise.

Noise-tolerant loss functions outperform standard losses in noisy ranking tasks.

Theoretical analysis confirms robustness of the proposed methods.

Abstract

Often, the data used to train ranking models is subject to label noise. For example, in web-search, labels created from clickstream data are noisy due to issues such as insufficient information in item descriptions on the SERP, query reformulation by the user, and erratic or unexpected user behavior. In practice, it is difficult to handle label noise without making strong assumptions about the label generation process. As a result, practitioners typically train their learning-to-rank (LtR) models directly on this noisy data without additional consideration of the label noise. Surprisingly, we often see strong performance from LtR models trained in this way. In this work, we describe a class of noise-tolerant LtR losses for which empirical risk minimization is a consistent procedure, even in the context of class-conditional label noise. We also develop noise-tolerant analogs of commonly used loss functions. The practical implications of our theoretical findings are further supported by experimental results.