A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks

TL;DR

This paper presents SFID, a versatile debiasing method for Vision-Language Models that reduces societal biases across multiple tasks without retraining, maintaining performance and enhancing fairness.

Contribution

Introduces SFID, a novel debiasing technique combining feature pruning and low confidence imputation, applicable across various VLM tasks without retraining.

Findings

Significantly reduces gender biases in VLMs

Maintains model performance across tasks

Applicable to multiple VLM architectures

Abstract

Recent advancements in Vision-Language Models (VLMs) have enabled complex multimodal tasks by processing text and image data simultaneously, significantly enhancing the field of artificial intelligence. However, these models often exhibit biases that can skew outputs towards societal stereotypes, thus necessitating debiasing strategies. Existing debiasing methods focus narrowly on specific modalities or tasks, and require extensive retraining. To address these limitations, this paper introduces Selective Feature Imputation for Debiasing (SFID), a novel methodology that integrates feature pruning and low confidence imputation (LCI) to effectively reduce biases in VLMs. SFID is versatile, maintaining the semantic integrity of outputs and costly effective by eliminating the need for retraining. Our experimental results demonstrate SFID's effectiveness across various VLMs tasks including zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation, by significantly reducing gender biases without compromising performance. This approach not only enhances the fairness of VLMs applications but also preserves their efficiency and utility across diverse scenarios.