Training-Free Test-Time Adaptation with Brownian Distance Covariance in Vision-Language Models

TL;DR

This paper introduces TaTa, a training-free, efficient test-time adaptation method for vision-language models that uses Brownian Distance Covariance to improve domain generalization without retraining.

Contribution

It proposes a novel, training-free adaptation approach leveraging Brownian Distance Covariance, enhancing efficiency and stability in vision-language models under domain shift.

Findings

Significantly reduces computational cost compared to existing methods.

Achieves state-of-the-art performance in domain and cross-dataset generalization.

Improves model stability by avoiding weight updates.

Abstract

Vision-language models suffer performance degradation under domain shift, limiting real-world applicability. Existing test-time adaptation methods are computationally intensive, rely on back-propagation, and often focus on single modalities. To address these issues, we propose Training-free Test-Time Adaptation with Brownian Distance Covariance (TaTa). TaTa leverages Brownian Distance Covariance-a powerful statistical measure that captures both linear and nonlinear dependencies via pairwise distances-to dynamically adapt VLMs to new domains without training or back-propagation. This not only improves efficiency but also enhances stability by avoiding disruptive weight updates. TaTa further integrates attribute-enhanced prompting to improve vision-language inference with descriptive visual cues. Combined with dynamic clustering and pseudo-label refinement, it effectively recalibrates the model for novel visual contexts. Experiments across diverse datasets show that TaTa significantly reduces computational cost while achieving state-of-the-art performance in domain and cross-dataset generalization.