SAGE: Spuriousness-Aware Guided Prompt Exploration for Mitigating Multimodal Bias

TL;DR

This paper introduces SAGE, a prompt exploration method that reduces multimodal spurious bias in vision-language models like CLIP, enhancing robustness and zero-shot performance without additional training or annotations.

Contribution

SAGE is a novel, training-free prompt selection technique that mitigates spurious biases in multimodal models by maximizing semantic separation between classes.

Findings

SAGE improves zero-shot classification robustness across multiple datasets.

It outperforms previous methods without requiring fine-tuning or external data.

SAGE is effective across various backbone models.

Abstract

Large vision-language models, such as CLIP, have shown strong zero-shot classification performance by aligning images and text in a shared embedding space. However, CLIP models often develop multimodal spurious biases, which is the undesirable tendency to rely on spurious features. For example, CLIP may infer object types in images based on frequently co-occurring backgrounds rather than the object's core features. This bias significantly impairs the robustness of pre-trained CLIP models on out-of-distribution data, where such cross-modal associations no longer hold. Existing methods for mitigating multimodal spurious bias typically require fine-tuning on downstream data or prior knowledge of the bias, which undermines the out-of-the-box usability of CLIP. In this paper, we first theoretically analyze the impact of multimodal spurious bias in zero-shot classification. Based on this insight, we propose Spuriousness-Aware Guided Exploration (SAGE), a simple and effective method that mitigates spurious bias through guided prompt selection. SAGE requires no training, fine-tuning, or external annotations. It explores a space of prompt templates and selects the prompts that induce the largest semantic separation between classes, thereby improving worst-group robustness. Extensive experiments on four real-world benchmark datasets and five popular backbone models demonstrate that SAGE consistently improves zero-shot performance and generalization, outperforming previous zero-shot approaches without any external knowledge or model updates.