Robustness and uncertainty: two complementary aspects of the reliability of the predictions of a classifier

TL;DR

This paper compares robustness and uncertainty quantification methods for classifier prediction reliability, demonstrating their complementarity and proposing a hybrid approach that improves overall assessment accuracy.

Contribution

It introduces a hybrid method combining robustness and uncertainty quantification, outperforming individual approaches in reliability assessment.

Findings

Hybrid approach outperforms individual methods

Both methods are complementary in assessing reliability

Relative importance of uncertainty and robustness varies by dataset

Abstract

We consider two conceptually different approaches for assessing the reliability of the individual predictions of a classifier: Robustness Quantification (RQ) and Uncertainty Quantification (UQ). We compare both approaches on a number of benchmark datasets and show that there is no clear winner between the two, but that they are complementary and can be combined to obtain a hybrid approach that outperforms both RQ and UQ. As a byproduct of our approach, for each dataset, we also obtain an assessment of the relative importance of uncertainty and robustness as sources of unreliability.