Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptation

TL;DR

This paper proposes a novel method for few-shot test-time domain adaptation by learning directly on input space and enhancing text feature diversity, significantly improving CLIP's performance on real-world benchmarks.

Contribution

It introduces a new approach that complements CLIP's frozen features with input space learning and dataset-specific text refinement for better domain adaptation.

Findings

Outperforms state-of-the-art on 5 large-scale benchmarks

Significantly improves performance of smaller networks like ViT-B/16

Achieves +5.1 F1 score on iWildCam and +3.1% WC accuracy on FMoW

Abstract

Few-shot Test-Time Domain Adaptation focuses on adapting a model at test time to a specific domain using only a few unlabeled examples, addressing domain shift. Prior methods leverage CLIP's strong out-of-distribution (OOD) abilities by generating domain-specific prompts to guide its generalized, frozen features. However, since downstream datasets are not explicitly seen by CLIP, solely depending on the feature space knowledge is constrained by CLIP's prior knowledge. Notably, when using a less robust backbone like ViT-B/16, performance significantly drops on challenging real-world benchmarks. Departing from the state-of-the-art of inheriting the intrinsic OOD capability of CLIP, this work introduces learning directly on the input space to complement the dataset-specific knowledge for frozen CLIP. Specifically, an independent side branch is attached in parallel with CLIP and enforced to learn exclusive knowledge via revert attention. To better capture the dataset-specific label semantics for downstream adaptation, we propose to enhance the inter-dispersion among text features via greedy text ensemble and refinement. The text and visual features are then progressively fused in a domain-aware manner by a generated domain prompt to adapt toward a specific domain. Extensive experiments show our method's superiority on 5 large-scale benchmarks (WILDS and DomainNet), notably improving over smaller networks like ViT-B/16 with gains of \textbf{+5.1} in F1 for iWildCam and \textbf{+3.1\%} in WC Acc for FMoW.