Dynamic Multimodal Prototype Learning in Vision-Language Models

TL;DR

ProtoMM is a training-free, multimodal prototype learning framework that dynamically adapts vision-language models during testing by combining textual and visual features, improving zero-shot classification accuracy.

Contribution

It introduces a novel multimodal prototype approach that dynamically updates during test time, addressing ambiguity in class descriptions and enhancing model generalization.

Findings

Achieves 1.03% average accuracy improvement on ImageNet benchmarks.

Effectively combines textual and visual features for prototype learning.

Demonstrates robustness across 15 zero-shot benchmarks.

Abstract

With the increasing attention to pre-trained vision-language models (VLMs), \eg, CLIP, substantial efforts have been devoted to many downstream tasks, especially in test-time adaptation (TTA). However, previous works focus on learning prototypes only in the textual modality while overlooking the ambiguous semantics in class names. These ambiguities lead to textual prototypes that are insufficient to capture visual concepts, resulting in limited performance. To address this issue, we introduce \textbf{ProtoMM}, a training-free framework that constructs multimodal prototypes to adapt VLMs during the test time. By viewing the prototype as a discrete distribution over the textual descriptions and visual particles, ProtoMM has the ability to combine the multimodal features for comprehensive prototype learning. More importantly, the visual particles are dynamically updated as the testing stream flows. This allows our multimodal prototypes to continually learn from the data, enhancing their generalizability in unseen scenarios. In addition, we quantify the importance of the prototypes and test images by formulating their semantic distance as an optimal transport problem. Extensive experiments on 15 zero-shot benchmarks demonstrate the effectiveness of our method, achieving a 1.03\% average accuracy improvement over state-of-the-art methods on ImageNet and its variant datasets.