Quantitative robustness of instance ranking problems

TL;DR

This paper introduces a new robustness measure called the order-inversal breakdown point (OIBDP) for ranking estimators, analyzing its properties and bounds in linear models and discussing extensions to SVM-based ranking methods.

Contribution

It defines the OIBDP for ranking problems, providing theoretical analysis, characterizations, and bounds, filling a gap in robustness measures for ranking estimators.

Findings

OIBDP measures the fraction of data needed to invert the ranking

Characterizations of the OIBDP for linear models are provided

Asymptotic upper bounds for the OIBDP are established

Abstract

Instance ranking problems intend to recover the true ordering of the instances in a data set with a variety of applications in for example scientific, social and financial contexts. Robust statistics studies the behaviour of estimators in the presence of perturbations of the data resp. the underlying distribution and provides different concepts to characterize local and global robustness. In this work, we concentrate on the global robustness of parametric ranking problems in terms of the breakdown point which measures the fraction of samples that need to be perturbed in order to let the estimator take unreasonable values. However, existing breakdown point notions do not cover ranking problems so far. We propose to define a breakdown of the estimator as a sign-reversal of all components which causes the predicted ranking to be potentially completely inverted, therefore we call our concept the order-inversal breakdown point (OIBDP). We will study the OIBDP, based on a linear model, for several different ranking problems that we carefully distinguish and provide least favorable outlier configurations, characterizations of the order-inversal breakdown point as well as sharp asymptotic upper bounds. We also outline the case of SVM-type ranking estimators.