Improving self-training under distribution shifts via anchored confidence with theoretical guarantees

TL;DR

This paper introduces a theoretically grounded method using anchored confidence and temporal ensembles to enhance self-training under distribution shifts, achieving significant accuracy improvements without extra computational costs.

Contribution

It proposes a novel uncertainty-aware temporal ensemble approach with theoretical guarantees to improve self-training under distribution shifts.

Findings

Improves self-training accuracy by 8-16% across various shifts

Enhances calibration and robustness to hyperparameters

Provides theoretical guarantees for asymptotic correctness

Abstract

Self-training often falls short under distribution shifts due to an increased discrepancy between prediction confidence and actual accuracy. This typically necessitates computationally demanding methods such as neighborhood or ensemble-based label corrections. Drawing inspiration from insights on early learning regularization, we develop a principled method to improve self-training under distribution shifts based on temporal consistency. Specifically, we build an uncertainty-aware temporal ensemble with a simple relative thresholding. Then, this ensemble smooths noisy pseudo labels to promote selective temporal consistency. We show that our temporal ensemble is asymptotically correct and our label smoothing technique can reduce the optimality gap of self-training. Our extensive experiments validate that our approach consistently improves self-training performances by 8% to 16% across diverse distribution shift scenarios without a computational overhead. Besides, our method exhibits attractive properties, such as improved calibration performance and robustness to different hyperparameter choices.