SAGE: Spuriousness-Aware Guided Prompt Exploration for Mitigating Multimodal Bias
Wenqian Ye, Di Wang, Guangtao Zheng, Bohan Liu, Aidong Zhang

TL;DR
This paper introduces SAGE, a prompt exploration method that reduces multimodal spurious bias in vision-language models like CLIP, enhancing robustness and zero-shot performance without additional training or annotations.
Contribution
SAGE is a novel, training-free prompt selection technique that mitigates spurious biases in multimodal models by maximizing semantic separation between classes.
Findings
SAGE improves zero-shot classification robustness across multiple datasets.
It outperforms previous methods without requiring fine-tuning or external data.
SAGE is effective across various backbone models.
Abstract
Large vision-language models, such as CLIP, have shown strong zero-shot classification performance by aligning images and text in a shared embedding space. However, CLIP models often develop multimodal spurious biases, which is the undesirable tendency to rely on spurious features. For example, CLIP may infer object types in images based on frequently co-occurring backgrounds rather than the object's core features. This bias significantly impairs the robustness of pre-trained CLIP models on out-of-distribution data, where such cross-modal associations no longer hold. Existing methods for mitigating multimodal spurious bias typically require fine-tuning on downstream data or prior knowledge of the bias, which undermines the out-of-the-box usability of CLIP. In this paper, we first theoretically analyze the impact of multimodal spurious bias in zero-shot classification. Based on this…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications · Adversarial Robustness in Machine Learning
