Ranking the best instances

TL;DR

This paper introduces a novel local ranking framework focusing on identifying the best instances using bipartite ranking, empirical risk minimization, and specialized performance measures extending AUC.

Contribution

It develops a new methodology for local ranking based on real-valued scoring functions and introduces performance measures tailored for ranking the top instances.

Findings

Extended AUC criteria for local ranking

Optimal elements of new ranking criteria identified

Preliminary statistical results established

Abstract

We formulate the local ranking problem in the framework of bipartite ranking where the goal is to focus on the best instances. We propose a methodology based on the construction of real-valued scoring functions. We study empirical risk minimization of dedicated statistics which involve empirical quantiles of the scores. We first state the problem of finding the best instances which can be cast as a classification problem with mass constraint. Next, we develop special performance measures for the local ranking problem which extend the Area Under an ROC Curve (AUC/AROC) criterion and describe the optimal elements of these new criteria. We also highlight the fact that the goal of ranking the best instances cannot be achieved in a stage-wise manner where first, the best instances would be tentatively identified and then a standard AUC criterion could be applied. Eventually, we state preliminary statistical results for the local ranking problem.