Unleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptation

TL;DR

This paper introduces MS-TTA, a training-free test-time adaptation method that refines all test sample features using mean-shift, significantly improving the robustness of visual-language models like CLIP under distribution shifts.

Contribution

MS-TTA is a novel, training-free approach that enhances feature representations beyond the original space using a single-step kNN mean-shift, improving adaptation stability and performance.

Findings

Outperforms state-of-the-art TTA methods on OOD benchmarks

Enhances feature compactness and class separability

Achieves robust adaptation without extra training

Abstract

Visual-language models (VLMs) like CLIP exhibit strong generalization but struggle with distribution shifts at test time. Existing training-free test-time adaptation (TTA) methods operate strictly within CLIP's original feature space, relying on high-confidence samples while overlooking the potential of low-confidence ones. We propose MS-TTA, a training-free approach that enhances feature representations beyond CLIP's space using a single-step k-nearest neighbors (kNN) Mean-Shift. By refining all test samples, MS-TTA improves feature compactness and class separability, leading to more stable adaptation. Additionally, a cache of refined embeddings further enhances inference by providing Mean Shift enhanced logits. Extensive evaluations on OOD and cross-dataset benchmarks demonstrate that MS-TTA consistently outperforms state-of-the-art training-free TTA methods, achieving robust adaptation without requiring additional training.