A Self-boosted Framework for Calibrated Ranking

TL;DR

This paper introduces SBCR, a novel framework for calibrated ranking that addresses limitations of existing multi-objective methods by improving training effectiveness and balancing calibration with ranking.

Contribution

The paper proposes SBCR, a self-boosted framework that enhances calibrated ranking by overcoming data aggregation issues and conflicting loss functions in existing methods.

Findings

Improved ranking calibration accuracy.

Enhanced training efficiency in online applications.

Better trade-off between calibration and ranking quality.

Abstract

Scale-calibrated ranking systems are ubiquitous in real-world applications nowadays, which pursue accurate ranking quality and calibrated probabilistic predictions simultaneously. For instance, in the advertising ranking system, the predicted click-through rate (CTR) is utilized for ranking and required to be calibrated for the downstream cost-per-click ads bidding. Recently, multi-objective based methods have been wildly adopted as a standard approach for Calibrated Ranking, which incorporates the combination of two loss functions: a pointwise loss that focuses on calibrated absolute values and a ranking loss that emphasizes relative orderings. However, when applied to industrial online applications, existing multi-objective CR approaches still suffer from two crucial limitations. First, previous methods need to aggregate the full candidate list within a single mini-batch to compute the ranking loss. Such aggregation strategy violates extensive data shuffling which has long been proven beneficial for preventing overfitting, and thus degrades the training effectiveness. Second, existing multi-objective methods apply the two inherently conflicting loss functions on a single probabilistic prediction, which results in a sub-optimal trade-off between calibration and ranking. To tackle the two limitations, we propose a Self-Boosted framework for Calibrated Ranking (SBCR).