Test-time Batch Statistics Calibration for Covariate Shift

TL;DR

This paper introduces a novel test-time adaptation method called $oldsymbol{ extalpha}$-BN that calibrates batch statistics during inference to improve neural network performance under covariate shift without additional training.

Contribution

The paper proposes $oldsymbol{ extalpha}$-BN, a new approach for test-time adaptation that mixes source and target batch statistics to handle covariate shift effectively.

Findings

Achieves state-of-the-art results on 12 datasets across various tasks.

Improves performance by 28.4% to 43.9% on GTA5 to Cityscapes without training.

Outperforms recent source-free domain adaptation methods.

Abstract

Deep neural networks have a clear degradation when applying to the unseen environment due to the covariate shift. Conventional approaches like domain adaptation requires the pre-collected target data for iterative training, which is impractical in real-world applications. In this paper, we propose to adapt the deep models to the novel environment during inference. An previous solution is test time normalization, which substitutes the source statistics in BN layers with the target batch statistics. However, we show that test time normalization may potentially deteriorate the discriminative structures due to the mismatch between target batch statistics and source parameters. To this end, we present a general formulation $\alpha$-BN to calibrate the batch statistics by mixing up the source and target statistics for both alleviating the domain shift and preserving the discriminative structures. Based on $\alpha$-BN, we further present a novel loss function to form a unified test time adaptation framework Core, which performs the pairwise class correlation online optimization. Extensive experiments show that our approaches achieve the state-of-the-art performance on total twelve datasets from three topics, including model robustness to corruptions, domain generalization on image classification and semantic segmentation. Particularly, our $\alpha$-BN improves 28.4\% to 43.9\% on GTA5 $\rightarrow$ Cityscapes without any training, even outperforms the latest source-free domain adaptation method.