Uncertainty Calibration with Energy Based Instance-wise Scaling in the Wild Dataset

TL;DR

This paper proposes a novel energy-based instance-wise calibration method to improve the reliability of uncertainty estimates in deep neural networks, especially for out-of-distribution data, enhancing safety in real-world AI applications.

Contribution

The paper introduces an energy model-based calibration technique that adapts uncertainty estimates per input, outperforming existing methods in diverse scenarios including OOD detection.

Findings

Consistently robust calibration across in-distribution and OOD data

Outperforms state-of-the-art calibration methods

Improves uncertainty estimation accuracy in real-world scenarios

Abstract

With the rapid advancement in the performance of deep neural networks (DNNs), there has been significant interest in deploying and incorporating artificial intelligence (AI) systems into real-world scenarios. However, many DNNs lack the ability to represent uncertainty, often exhibiting excessive confidence even when making incorrect predictions. To ensure the reliability of AI systems, particularly in safety-critical cases, DNNs should transparently reflect the uncertainty in their predictions. In this paper, we investigate robust post-hoc uncertainty calibration methods for DNNs within the context of multi-class classification tasks. While previous studies have made notable progress, they still face challenges in achieving robust calibration, particularly in scenarios involving out-of-distribution (OOD). We identify that previous methods lack adaptability to individual input data and struggle to accurately estimate uncertainty when processing inputs drawn from the wild dataset. To address this issue, we introduce a novel instance-wise calibration method based on an energy model. Our method incorporates energy scores instead of softmax confidence scores, allowing for adaptive consideration of DNN uncertainty for each prediction within a logit space. In experiments, we show that the proposed method consistently maintains robust performance across the spectrum, spanning from in-distribution to OOD scenarios, when compared to other state-of-the-art methods.