Robustness Quantification for Discriminative Models: a New Robustness Metric and its Application to Dynamic Classifier Selection

TL;DR

This paper introduces a new robustness metric for probabilistic discriminative classifiers that assesses prediction reliability without restrictive assumptions, enhancing dynamic classifier selection strategies.

Contribution

The authors propose a universally applicable robustness metric for probabilistic classifiers, overcoming limitations of previous methods that relied on generative models or specific features.

Findings

The new metric effectively distinguishes reliable from unreliable predictions.

It enables improved dynamic classifier selection strategies.

The approach applies to any probabilistic discriminative classifier.

Abstract

Among the different possible strategies for evaluating the reliability of individual predictions of classifiers, robustness quantification stands out as a method that evaluates how much uncertainty a classifier could cope with before changing its prediction. However, its applicability is more limited than some of its alternatives, since it requires the use of generative models and restricts the analyses either to specific model architectures or discrete features. In this work, we propose a new robustness metric applicable to any probabilistic discriminative classifier and any type of features. We demonstrate that this new metric is capable of distinguishing between reliable and unreliable predictions, and use this observation to develop new strategies for dynamic classifier selection.