Beyond In-Domain Scenarios: Robust Density-Aware Calibration

TL;DR

This paper introduces DAC, a density-aware calibration method that enhances uncertainty estimates of deep neural networks, especially under domain-shift and out-of-domain conditions, by leveraging hidden layer information.

Contribution

DAC is a novel, generic calibration approach that improves robustness in domain-shift and OOD scenarios while preserving in-domain accuracy, compatible with existing methods.

Findings

DAC improves calibration robustness across multiple architectures and datasets.

DAC enhances uncertainty estimates on large-scale pre-trained neural networks.

DAC maintains high in-domain calibration performance.

Abstract

Calibrating deep learning models to yield uncertainty-aware predictions is crucial as deep neural networks get increasingly deployed in safety-critical applications. While existing post-hoc calibration methods achieve impressive results on in-domain test datasets, they are limited by their inability to yield reliable uncertainty estimates in domain-shift and out-of-domain (OOD) scenarios. We aim to bridge this gap by proposing DAC, an accuracy-preserving as well as Density-Aware Calibration method based on k-nearest-neighbors (KNN). In contrast to existing post-hoc methods, we utilize hidden layers of classifiers as a source for uncertainty-related information and study their importance. We show that DAC is a generic method that can readily be combined with state-of-the-art post-hoc methods. DAC boosts the robustness of calibration performance in domain-shift and OOD, while maintaining excellent in-domain predictive uncertainty estimates. We demonstrate that DAC leads to consistently better calibration across a large number of model architectures, datasets, and metrics. Additionally, we show that DAC improves calibration substantially on recent large-scale neural networks pre-trained on vast amounts of data.