Ranking with Abstention

TL;DR

This paper introduces a new ranking framework allowing models to abstain at a cost, providing state-of-the-art theoretical guarantees and demonstrating practical effectiveness through experiments.

Contribution

It develops a comprehensive theoretical analysis with the best existing consistency bounds for ranking with abstention, applicable to linear and neural network models.

Findings

State-of-the-art $H$-consistency bounds established.

Abstention improves ranking performance in practice.

Experimental results confirm effectiveness of the proposed methods.

Abstract

We introduce a novel framework of ranking with abstention, where the learner can abstain from making prediction at some limited cost $c$. We present a extensive theoretical analysis of this framework including a series of $H$-consistency bounds for both the family of linear functions and that of neural networks with one hidden-layer. These theoretical guarantees are the state-of-the-art consistency guarantees in the literature, which are upper bounds on the target loss estimation error of a predictor in a hypothesis set $H$, expressed in terms of the surrogate loss estimation error of that predictor. We further argue that our proposed abstention methods are important when using common equicontinuous hypothesis sets in practice. We report the results of experiments illustrating the effectiveness of ranking with abstention.