Enhancing Test Time Adaptation with Few-shot Guidance

TL;DR

This paper introduces Few-Shot Test Time Adaptation (FS-TTA), a method that improves domain shift correction in neural networks using minimal support data, enhancing robustness and performance in real-world applications.

Contribution

The paper proposes a novel FS-TTA setting and a two-stage framework that combines fine-tuning with prototype memory guidance for effective domain adaptation.

Findings

FS-TTA outperforms existing TTA methods on multiple benchmarks.

The framework reduces overfitting through feature diversity augmentation.

Experimental results show significant accuracy improvements.

Abstract

Deep neural networks often encounter significant performance drops while facing with domain shifts between training (source) and test (target) data. To address this issue, Test Time Adaptation (TTA) methods have been proposed to adapt pre-trained source model to handle out-of-distribution streaming target data. Although these methods offer some relief, they lack a reliable mechanism for domain shift correction, which can often be erratic in real-world applications. In response, we develop Few-Shot Test Time Adaptation (FS-TTA), a novel and practical setting that utilizes a few-shot support set on top of TTA. Adhering to the principle of few inputs, big gains, FS-TTA reduces blind exploration in unseen target domains. Furthermore, we propose a two-stage framework to tackle FS-TTA, including (i) fine-tuning the pre-trained source model with few-shot support set, along with using feature diversity augmentation module to avoid overfitting, (ii) implementing test time adaptation based on prototype memory bank guidance to produce high quality pseudo-label for model adaptation. Through extensive experiments on three cross-domain classification benchmarks, we demonstrate the superior performance and reliability of our FS-TTA and framework.