Covariance-aware Feature Alignment with Pre-computed Source Statistics for Test-time Adaptation to Multiple Image Corruptions

TL;DR

This paper introduces CAFe, a test-time adaptation method that effectively handles multiple image corruptions by using covariance-aware feature alignment, improving robustness without retraining.

Contribution

The paper proposes a novel covariance-aware feature alignment method for test-time adaptation that better manages multiple corruptions in image recognition tasks.

Findings

CAFe outperforms prior TTA methods on multiple image corruptions.

It effectively reduces distribution gaps caused by diverse corruptions.

The method enhances robustness without retraining the model.

Abstract

Real-world image recognition systems often face corrupted input images, which cause distribution shifts and degrade the performance of models. These systems often use a single prediction model in a central server and process images sent from various environments, such as cameras distributed in cities or cars. Such single models face images corrupted in heterogeneous ways in test time. Thus, they require to instantly adapt to the multiple corruptions during testing rather than being re-trained at a high cost. Test-time adaptation (TTA), which aims to adapt models without accessing the training dataset, is one of the settings that can address this problem. Existing TTA methods indeed work well on a single corruption. However, the adaptation ability is limited when multiple types of corruption occur, which is more realistic. We hypothesize this is because the distribution shift is more complicated, and the adaptation becomes more difficult in case of multiple corruptions. In fact, we experimentally found that a larger distribution gap remains after TTA. To address the distribution gap during testing, we propose a novel TTA method named Covariance-Aware Feature alignment (CAFe). We empirically show that CAFe outperforms prior TTA methods on image corruptions, including multiple types of corruptions.