PaRCE: Probabilistic and Reconstruction-based Competency Estimation for CNN-based Image Classification

TL;DR

PaRCE introduces a probabilistic and reconstruction-based method to accurately estimate CNN model confidence, detect out-of-distribution inputs, and localize anomalies, providing a holistic uncertainty quantification approach.

Contribution

The paper presents a novel holistic uncertainty estimation method, PaRCE, that outperforms existing techniques in distinguishing correct, incorrect, and OOD samples with interpretable scores.

Findings

PaRCE best distinguishes between correct, misclassified, and OOD samples.

The method effectively localizes anomalous regions in images.

PaRCE provides reliable, interpretable confidence scores.

Abstract

Convolutional neural networks (CNNs) are extremely popular and effective for image classification tasks but tend to be overly confident in their predictions. Various works have sought to quantify uncertainty associated with these models, detect out-of-distribution (OOD) inputs, or identify anomalous regions in an image, but limited work has sought to develop a holistic approach that can accurately estimate perception model confidence across various sources of uncertainty. We develop a probabilistic and reconstruction-based competency estimation (PaRCE) method and compare it to existing approaches for uncertainty quantification and OOD detection. We find that our method can best distinguish between correctly classified, misclassified, and OOD samples with anomalous regions, as well as between samples with visual image modifications resulting in high, medium, and low prediction accuracy. We describe how to extend our approach for anomaly localization tasks and demonstrate the ability of our approach to distinguish between regions in an image that are familiar to the perception model from those that are unfamiliar. We find that our method generates interpretable scores that most reliably capture a holistic notion of perception model confidence.