Subspace Alignment for Vision-Language Model Test-time Adaptation

TL;DR

SubTTA improves vision-language model test-time adaptation by aligning semantic subspaces of visual and textual modalities, effectively addressing distribution shifts and visual noise, leading to better zero-shot guidance and performance.

Contribution

The paper introduces SubTTA, a novel method that aligns semantic subspaces of modalities and filters visual noise, enhancing test-time adaptation of VLMs under distribution shifts.

Findings

SubTTA achieves an average of 2.24% improvement over state-of-the-art TTA methods.

Aligning semantic subspaces reduces modality gap and visual nuisance.

Extensive experiments validate the effectiveness across benchmarks and architectures.

Abstract

Vision-language models (VLMs), despite their extraordinary zero-shot capabilities, are vulnerable to distribution shifts. Test-time adaptation (TTA) emerges as a predominant strategy to adapt VLMs to unlabeled test data on the fly. However, existing TTA methods heavily rely on zero-shot predictions as pseudo-labels for self-training, which can be unreliable under distribution shifts and misguide adaptation due to two fundamental limitations. First (Modality Gap), distribution shifts induce gaps between visual and textual modalities, making cross-modal relations inaccurate. Second (Visual Nuisance), visual embeddings encode rich but task-irrelevant noise that often overwhelms task-specific semantics under distribution shifts. To address these limitations, we propose SubTTA, which aligns the semantic subspaces of both modalities to enhance zero-shot predictions to better guide the TTA process. To bridge the modality gap, SubTTA extracts the principal subspaces of both modalities and aligns the visual manifold to the textual semantic anchor by minimizing their chordal distance. To eliminate visual nuisance, SubTTA projects the aligned visual features onto the task-specific textual subspace, which filters out task-irrelevant noise by constraining visual embeddings within the valid semantic span, and standard TTA is further performed on the purified space to refine the decision boundaries. Extensive experiments on various benchmarks and VLM architectures demonstrate the effectiveness of SubTTA, yielding an average improvement of 2.24% over state-of-the-art TTA methods.