Interpretable Debiasing of Vision-Language Models for Social Fairness

TL;DR

This paper introduces DeBiasLens, an interpretable framework that localizes and deactivates social attribute neurons in vision-language models to mitigate bias while preserving semantic understanding.

Contribution

It presents a novel, model-agnostic bias mitigation method using sparse autoencoders to identify and deactivate social bias neurons without needing labeled social attributes.

Findings

Effective bias mitigation by deactivating social neurons

Preserves semantic knowledge of models

Supports future fairness auditing tools

Abstract

The rapid advancement of Vision-Language models (VLMs) has raised growing concerns that their black-box reasoning processes could lead to unintended forms of social bias. Current debiasing approaches focus on mitigating surface-level bias signals through post-hoc learning or test-time algorithms, while leaving the internal dynamics of the model largely unexplored. In this work, we introduce an interpretable, model-agnostic bias mitigation framework, DeBiasLens, that localizes social attribute neurons in VLMs through sparse autoencoders (SAEs) applied to multimodal encoders. Building upon the disentanglement ability of SAEs, we train them on facial image or caption datasets without corresponding social attribute labels to uncover neurons highly responsive to specific demographics, including those that are underrepresented. By selectively deactivating the social neurons most strongly tied to bias for each group, we effectively mitigate socially biased behaviors of VLMs without degrading their semantic knowledge. Our research lays the groundwork for future auditing tools, prioritizing social fairness in emerging real-world AI systems.