PseudoCal: A Source-Free Approach to Unsupervised Uncertainty Calibration in Domain Adaptation

TL;DR

PseudoCal introduces a novel source-free method for calibrating predictive uncertainty in unsupervised domain adaptation, leveraging pseudo-target data to improve calibration accuracy without requiring source data.

Contribution

The paper proposes PseudoCal, a new source-free calibration approach that transforms unsupervised calibration into a supervised problem using pseudo-target data, outperforming existing methods.

Findings

PseudoCal achieves significantly lower calibration error across 10 UDA methods.

It performs well in both traditional and source-free UDA scenarios.

Extensive experiments validate its effectiveness over existing calibration techniques.

Abstract

Unsupervised domain adaptation (UDA) has witnessed remarkable advancements in improving the accuracy of models for unlabeled target domains. However, the calibration of predictive uncertainty in the target domain, a crucial aspect of the safe deployment of UDA models, has received limited attention. The conventional in-domain calibration method, \textit{temperature scaling} (TempScal), encounters challenges due to domain distribution shifts and the absence of labeled target domain data. Recent approaches have employed importance-weighting techniques to estimate the target-optimal temperature based on re-weighted labeled source data. Nonetheless, these methods require source data and suffer from unreliable density estimates under severe domain shifts, rendering them unsuitable for source-free UDA settings. To overcome these limitations, we propose PseudoCal, a source-free calibration method that exclusively relies on unlabeled target data. Unlike previous approaches that treat UDA calibration as a \textit{covariate shift} problem, we consider it as an unsupervised calibration problem specific to the target domain. Motivated by the factorization of the negative log-likelihood (NLL) objective in TempScal, we generate a labeled pseudo-target set that captures the structure of the real target. By doing so, we transform the unsupervised calibration problem into a supervised one, enabling us to effectively address it using widely-used in-domain methods like TempScal. Finally, we thoroughly evaluate the calibration performance of PseudoCal by conducting extensive experiments on 10 UDA methods, considering both traditional UDA settings and recent source-free UDA scenarios. The experimental results consistently demonstrate the superior performance of PseudoCal, exhibiting significantly reduced calibration error compared to existing calibration methods.