Label Distribution Shift-Aware Prediction Refinement for Test-Time Adaptation

TL;DR

This paper introduces DART, a novel test-time adaptation method that refines predictions by focusing on class-wise confusion patterns, significantly improving accuracy under label distribution shifts without degrading performance otherwise.

Contribution

DART is the first method to explicitly model and correct class-wise confusion patterns during test-time adaptation, enhancing robustness to label distribution shifts.

Findings

DART achieves 5-18% accuracy gains on CIFAR-10C.

DART improves performance on multiple benchmarks including CIFAR, PACS, OfficeHome, and ImageNet.

DART enhances existing TTA methods by correcting inaccurate predictions caused by distribution shifts.

Abstract

Test-time adaptation (TTA) is an effective approach to mitigate performance degradation of trained models when encountering input distribution shifts at test time. However, existing TTA methods often suffer significant performance drops when facing additional class distribution shifts. We first analyze TTA methods under label distribution shifts and identify the presence of class-wise confusion patterns commonly observed across different covariate shifts. Based on this observation, we introduce label Distribution shift-Aware prediction Refinement for Test-time adaptation (DART), a novel TTA method that refines the predictions by focusing on class-wise confusion patterns. DART trains a prediction refinement module during an intermediate time by exposing it to several batches with diverse class distributions using the training dataset. This module is then used during test time to detect and correct class distribution shifts, significantly improving pseudo-label accuracy for test data. Our method exhibits 5-18% gains in accuracy under label distribution shifts on CIFAR-10C, without any performance degradation when there is no label distribution shift. Extensive experiments on CIFAR, PACS, OfficeHome, and ImageNet benchmarks demonstrate DART's ability to correct inaccurate predictions caused by test-time distribution shifts. This improvement leads to enhanced performance in existing TTA methods, making DART a valuable plug-in tool.