Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition

TL;DR

This paper identifies the over-reliance of vision-language models on spuriously correlated attributes, proposing methods to filter and shield against these biases, thereby improving out-of-distribution generalization.

Contribution

It introduces Spurious Attribute Probing and Spurious Attribute Shielding to detect and mitigate biased attributes, enhancing model robustness without sacrificing downstream performance.

Findings

SAP and SAS improve accuracy on distribution shifts

Achieve state-of-the-art results across 11 datasets

Enhance generalization without harming downstream tasks

Abstract

Few-shot adaptation for Vision-Language Models (VLMs) presents a dilemma: balancing in-distribution accuracy with out-of-distribution generalization. Recent research has utilized low-level concepts such as visual attributes to enhance generalization. However, this study reveals that VLMs overly rely on a small subset of attributes on decision-making, which co-occur with the category but are not inherently part of it, termed spuriously correlated attributes. This biased nature of VLMs results in poor generalization. To address this, 1) we first propose Spurious Attribute Probing (SAP), identifying and filtering out these problematic attributes to significantly enhance the generalization of existing attribute-based methods; 2) We introduce Spurious Attribute Shielding (SAS), a plug-and-play module that mitigates the influence of these attributes on prediction, seamlessly integrating into various Parameter-Efficient Fine-Tuning (PEFT) methods. In experiments, SAP and SAS significantly enhance accuracy on distribution shifts across 11 datasets and 3 generalization tasks without compromising downstream performance, establishing a new state-of-the-art benchmark.