Regression Compatible Listwise Objectives for Calibrated Ranking with Binary Relevance

TL;DR

This paper introduces a regression compatible listwise ranking method that aligns ranking and regression objectives, improving score calibration and ranking quality in learning-to-rank systems, with successful deployment in YouTube.

Contribution

It proposes a novel RCR approach that mutually aligns ranking and regression objectives, enhancing multi-objective trade-offs and calibration in learning-to-rank models.

Findings

Achieves top or competitive results on public benchmarks.

Significantly improves Pareto frontiers in multi-objective optimization.

Enhances ranking and click prediction in YouTube Search.

Abstract

As Learning-to-Rank (LTR) approaches primarily seek to improve ranking quality, their output scores are not scale-calibrated by design. This fundamentally limits LTR usage in score-sensitive applications. Though a simple multi-objective approach that combines a regression and a ranking objective can effectively learn scale-calibrated scores, we argue that the two objectives are not necessarily compatible, which makes the trade-off less ideal for either of them. In this paper, we propose a practical regression compatible ranking (RCR) approach that achieves a better trade-off, where the two ranking and regression components are proved to be mutually aligned. Although the same idea applies to ranking with both binary and graded relevance, we mainly focus on binary labels in this paper. We evaluate the proposed approach on several public LTR benchmarks and show that it consistently achieves either best or competitive result in terms of both regression and ranking metrics, and significantly improves the Pareto frontiers in the context of multi-objective optimization. Furthermore, we evaluated the proposed approach on YouTube Search and found that it not only improved the ranking quality of the production pCTR model, but also brought gains to the click prediction accuracy. The proposed approach has been successfully deployed in the YouTube production system.