Learning Calibrated Uncertainties for Domain Shift: A Distributionally Robust Learning Approach

TL;DR

This paper introduces a distributionally robust learning framework that calibrates uncertainties under domain shifts by estimating density ratios, improving performance in domain adaptation and semi-supervised learning tasks.

Contribution

It presents a novel method combining density ratio estimation with task networks to calibrate uncertainties during domain shifts, enhancing robustness and downstream task performance.

Findings

Improved cross-domain performance in UDA and SSL tasks.

Density ratio estimates correlate with human uncertainty perceptions.

Method benefits pseudo-label selection in semi-supervised learning.

Abstract

We propose a framework for learning calibrated uncertainties under domain shifts, where the source (training) distribution differs from the target (test) distribution. We detect such domain shifts via a differentiable density ratio estimator and train it together with the task network, composing an adjusted softmax predictive form concerning domain shift. In particular, the density ratio estimation reflects the closeness of a target (test) sample to the source (training) distribution. We employ it to adjust the uncertainty of prediction in the task network. This idea of using the density ratio is based on the distributionally robust learning (DRL) framework, which accounts for the domain shift by adversarial risk minimization. We show that our proposed method generates calibrated uncertainties that benefit downstream tasks, such as unsupervised domain adaptation (UDA) and semi-supervised learning (SSL). On these tasks, methods like self-training and FixMatch use uncertainties to select confident pseudo-labels for re-training. Our experiments show that the introduction of DRL leads to significant improvements in cross-domain performance. We also show that the estimated density ratios align with human selection frequencies, suggesting a positive correlation with a proxy of human perceived uncertainties.