Evaluating and Boosting Uncertainty Quantification in Classification

TL;DR

This paper introduces a new evaluation metric, AUCCC, for uncertainty quantification in classification, and proposes a distillation method, UDist, to enhance UQ performance, validated on natural and medical images.

Contribution

It presents a unified, robust metric for UQ evaluation and a simple distillation scheme to improve UQ accuracy in classification tasks.

Findings

AUCCC effectively evaluates UQ models.

UDist consistently improves UQ performance.

Method outperforms baselines on diverse datasets.

Abstract

Emergence of artificial intelligence techniques in biomedical applications urges the researchers to pay more attention on the uncertainty quantification (UQ) in machine-assisted medical decision making. For classification tasks, prior studies on UQ are difficult to compare with each other, due to the lack of a unified quantitative evaluation metric. Considering that well-performing UQ models ought to know when the classification models act incorrectly, we design a new evaluation metric, area under Confidence-Classification Characteristic curves (AUCCC), to quantitatively evaluate the performance of the UQ models. AUCCC is threshold-free, robust to perturbation, and insensitive to the classification performance. We evaluate several UQ methods (e.g., max softmax output) with AUCCC to validate its effectiveness. Furthermore, a simple scheme, named Uncertainty Distillation (UDist), is developed to boost the UQ performance, where a confidence model is distilling the confidence estimated by deep ensembles. The proposed method is easy to implement; it consistently outperforms strong baselines on natural and medical image datasets in our experiments.