Robust Calibration with Multi-domain Temperature Scaling

TL;DR

This paper introduces multi-domain temperature scaling, a calibration method that leverages data from multiple domains to improve uncertainty quantification robustness under distribution shifts in machine learning models.

Contribution

The paper proposes a novel multi-domain temperature scaling approach that enhances calibration robustness during distribution shifts, addressing a key challenge in uncertainty quantification.

Findings

Outperforms existing calibration methods on benchmark datasets

Improves calibration accuracy under distribution shifts

Effective in both in-distribution and out-of-distribution settings

Abstract

Uncertainty quantification is essential for the reliable deployment of machine learning models to high-stakes application domains. Uncertainty quantification is all the more challenging when training distribution and test distribution are different, even the distribution shifts are mild. Despite the ubiquity of distribution shifts in real-world applications, existing uncertainty quantification approaches mainly study the in-distribution setting where the train and test distributions are the same. In this paper, we develop a systematic calibration model to handle distribution shifts by leveraging data from multiple domains. Our proposed method -- multi-domain temperature scaling -- uses the heterogeneity in the domains to improve calibration robustness under distribution shift. Through experiments on three benchmark data sets, we find our proposed method outperforms existing methods as measured on both in-distribution and out-of-distribution test sets.