A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift

TL;DR

This paper introduces Unsupervised Temperature Scaling (UTS), a calibration method that improves uncertainty estimates of deep neural networks under domain shift by using unlabeled test data, enhancing robustness without requiring labels.

Contribution

The paper presents UTS, a novel unsupervised calibration technique specifically designed to handle domain shift, utilizing a new loss function and unlabeled test samples.

Findings

UTS effectively calibrates models without labeled data.

UTS outperforms existing methods like TS, MC-dropout, SVI, and ensembles under domain shift.

Demonstrates robustness across various datasets and models.

Abstract

The uncertainty estimation is critical in real-world decision making applications, especially when distributional shift between the training and test data are prevalent. Many calibration methods in the literature have been proposed to improve the predictive uncertainty of DNNs which are generally not well-calibrated. However, none of them is specifically designed to work properly under domain shift condition. In this paper, we propose Unsupervised Temperature Scaling (UTS) as a robust calibration method to domain shift. It exploits unlabeled test samples instead of the training one to adjust the uncertainty prediction of deep models towards the test distribution. UTS utilizes a novel loss function, weighted NLL, which allows unsupervised calibration. We evaluate UTS on a wide range of model-datasets to show the possibility of calibration without labels and demonstrate the robustness of UTS compared to other methods (e.g., TS, MC-dropout, SVI, ensembles) in shifted domains.