Post-hoc Uncertainty Calibration for Domain Drift Scenarios

TL;DR

This paper investigates the failure of existing post-hoc calibration methods under domain shift and proposes a simple perturbation strategy to improve calibration accuracy in such scenarios.

Contribution

It reveals the limitations of current calibration methods under domain drift and introduces a perturbation-based approach to enhance calibration performance.

Findings

Existing methods are over-confident under domain shift.

Perturbation improves calibration accuracy.

Method is effective across various architectures and tasks.

Abstract

We address the problem of uncertainty calibration. While standard deep neural networks typically yield uncalibrated predictions, calibrated confidence scores that are representative of the true likelihood of a prediction can be achieved using post-hoc calibration methods. However, to date the focus of these approaches has been on in-domain calibration. Our contribution is two-fold. First, we show that existing post-hoc calibration methods yield highly over-confident predictions under domain shift. Second, we introduce a simple strategy where perturbations are applied to samples in the validation set before performing the post-hoc calibration step. In extensive experiments, we demonstrate that this perturbation step results in substantially better calibration under domain shift on a wide range of architectures and modelling tasks.