Robustness Quantification and Uncertainty Quantification: Comparing Two Methods for Assessing the Reliability of Classifier Predictions

TL;DR

This paper compares two methods, Robustness Quantification and Uncertainty Quantification, for evaluating classifier prediction reliability, demonstrating RQ's superior performance and the benefits of combining both approaches.

Contribution

It provides a detailed comparison of RQ and UQ, highlighting RQ's advantages and the complementary nature of both methods for reliability assessment.

Findings

RQ can outperform UQ in standard and distribution shift scenarios

Combining RQ and UQ yields improved reliability assessments

RQ is capable of competitive performance against UQ

Abstract

We consider two approaches for assessing the reliability of the individual predictions of a classifier: Robustness Quantification (RQ) and Uncertainty Quantification (UQ). We explain the conceptual differences between the two approaches, compare both approaches on a number of benchmark datasets and show that RQ is capable of outperforming UQ, both in a standard setting and in the presence of distribution shift. Beside showing that RQ can be competitive with UQ, we also demonstrate the complementarity of RQ and UQ by showing that a combination of both approaches can lead to even better reliability assessments.