Test-Time Distribution Normalization for Contrastively Learned Vision-language Models

TL;DR

This paper introduces Distribution Normalization (DN), a test-time method that aligns inference with the training objective of contrastive vision-language models like CLIP, improving performance without retraining.

Contribution

It proposes a novel test-time normalization technique that captures negative sample information during inference, enhancing model accuracy across tasks.

Findings

DN outperforms standard dot product in various tasks

No retraining or fine-tuning required for DN

Extensive experiments validate DN's effectiveness

Abstract

Advances in the field of vision-language contrastive learning have made it possible for many downstream applications to be carried out efficiently and accurately by simply taking the dot product between image and text representations. One of the most representative approaches proposed recently known as CLIP has garnered widespread adoption due to its effectiveness. CLIP is trained with an InfoNCE loss that takes into account both positive and negative samples to help learn a much more robust representation space. This paper reveals that the common downstream practice of taking a dot product is only a zeroth-order approximation of the optimization goal, resulting in a loss of information during test-time. Intuitively, since the model has been optimized based on the InfoNCE loss, test-time procedures should also be in alignment. The question lies in how one can retrieve any semblance of negative samples information during inference in a computationally efficient way. To this end, we propose Distribution Normalization (DN), where we approximate the mean representation of a batch of test samples and use such a mean to represent what would be analogous to negative samples in the InfoNCE loss. DN requires no retraining or fine-tuning and can be effortlessly applied during inference. Extensive experiments on a wide variety of downstream tasks exhibit a clear advantage of DN over the dot product on top of other existing test-time augmentation methods.