Noise is an Efficient Learner for Zero-Shot Vision-Language Models

TL;DR

This paper introduces Test-Time Noise Tuning (TNT), a novel method that improves zero-shot vision-language model adaptation by optimizing visual input noise, leading to better generalization and calibration across diverse distributions.

Contribution

The paper proposes TNT, a new test-time adaptation technique that optimizes learnable noise in visual inputs and enforces inter-view representation coherence for improved model robustness.

Findings

Achieves +7.38% on natural distribution benchmarks

Gains +0.80% on cross-dataset evaluations

Enhances model calibration and out-of-distribution handling

Abstract

Recently, test-time adaptation has garnered attention as a method for tuning models without labeled data. The conventional modus operandi for adapting pre-trained vision-language models (VLMs) during test-time primarily focuses on tuning learnable prompts; however, this approach overlooks potential distribution shifts in the visual representations themselves. In this work, we address this limitation by introducing Test-Time Noise Tuning (TNT), a novel method for handling unpredictable shifts in the visual space. TNT leverages, for the first time, a noise adaptation strategy that optimizes learnable noise directly in the visual input space, enabling adaptive feature learning from a single test sample. We further introduce a novel approach for inter-view representation alignment by explicitly enforcing coherence in embedding distances, ensuring consistent feature representations across views. Combined with scaled logits and confident view selection at inference, TNT substantially enhances VLM generalization and calibration, achieving average gains of +7.38% on natural distributions benchmark and +0.80% on cross-dataset evaluations over zero-shot CLIP. These improvements lay a strong foundation for adaptive out-of-distribution handling.