SEVA: Leveraging Single-Step Ensemble of Vicinal Augmentations for Test-Time Adaptation

TL;DR

SEVA introduces a novel single-step ensemble approach for test-time adaptation that leverages multiple augmentations efficiently, improving model robustness without increasing computational costs.

Contribution

The paper proposes a theoretical framework and practical method to optimize augmentation effects in a single step, enhancing TTA efficiency and effectiveness.

Findings

SEVA outperforms existing TTA methods across multiple benchmarks.

The upper bound loss improves sample selection and model robustness.

SEVA maintains real-time performance with minimal computational overhead.

Abstract

Test-Time adaptation (TTA) aims to enhance model robustness against distribution shifts through rapid model adaptation during inference. While existing TTA methods often rely on entropy-based unsupervised training and achieve promising results, the common practice of a single round of entropy training is typically unable to adequately utilize reliable samples, hindering adaptation efficiency. In this paper, we discover augmentation strategies can effectively unleash the potential of reliable samples, but the rapidly growing computational cost impedes their real-time application. To address this limitation, we propose a novel TTA approach named Single-step Ensemble of Vicinal Augmentations (SEVA), which can take advantage of data augmentations without increasing the computational burden. Specifically, instead of explicitly utilizing the augmentation strategy to generate new data, SEVA develops a theoretical framework to explore the impacts of multiple augmentations on model adaptation and proposes to optimize an upper bound of the entropy loss to integrate the effects of multiple rounds of augmentation training into a single step. Furthermore, we discover and verify that using the upper bound as the loss is more conducive to the selection mechanism, as it can effectively filter out harmful samples that confuse the model. Combining these two key advantages, the proposed efficient loss and a complementary selection strategy can simultaneously boost the potential of reliable samples and meet the stringent time requirements of TTA. The comprehensive experiments on various network architectures across challenging testing scenarios demonstrate impressive performances and the broad adaptability of SEVA. The code will be publicly available.