Debiasing Vison-Language Models with Text-Only Training

TL;DR

This paper introduces TOD, a text-only debiasing framework for vision-language models that uses a text-as-image paradigm and a multi-target prediction task to improve bias mitigation without requiring image data.

Contribution

The paper proposes a novel text-only debiasing method for VLMs that repurposes text encoders as image encoders and employs a multi-target prediction task to reduce bias.

Findings

Achieves state-of-the-art results among image-free debiasing methods.

Significantly improves group robustness on Waterbirds and CelebA datasets.

Demonstrates strong generalization to multiple and unknown bias attributes.

Abstract

Pre-trained vision-language models (VLMs), such as CLIP, have exhibited remarkable performance across various downstream tasks by aligning text and images in a unified embedding space. However, due to the imbalanced distribution of pre-trained datasets, CLIP suffers from the bias problem in real-world applications. Existing debiasing methods struggle to obtain sufficient image samples for minority groups and incur high costs for group labeling. To address the limitations, we propose a Text-Only Debiasing framework called TOD, leveraging a text-as-image training paradigm to mitigate visual biases. Specifically, this approach repurposes the text encoder to function as an image encoder, thereby eliminating the need for image data. Simultaneously, it utilizes a large language model (LLM) to generate a balanced text dataset, which is then used for prompt tuning. However, we observed that the model overfits to the text modality because label names, serving as supervision signals, appear explicitly in the texts. To address this issue, we further introduce a Multi-Target Prediction (MTP) task that motivates the model to focus on complex contexts and distinguish between target and biased information. Extensive experiments on the Waterbirds and CelebA datasets show that our method significantly improves group robustness, achieving state-of-the-art results among image-free methods and even competitive performance compared to image-supervised methods. Furthermore, the proposed method can be adapted to challenging scenarios with multiple or unknown bias attributes, demonstrating its strong generalization and robustness.