Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluation

TL;DR

This paper investigates the origins of social bias in Large Vision Language Models by analyzing information flow and multi-round dialogues, revealing biases in internal reasoning and semantic representations across modalities.

Contribution

It introduces an explanatory framework combining information flow analysis with dialogue evaluation to understand bias mechanisms in LVLMs.

Findings

LVLMs show systematic disparities in information usage across demographic groups.

Bias is rooted in internal reasoning dynamics of the models.

Semantic representations also exhibit biased proximity patterns.

Abstract

Large Vision Language Models (LVLMs) have achieved remarkable progress in multimodal tasks, yet they also exhibit notable social biases. These biases often manifest as unintended associations between neutral concepts and sensitive human attributes, leading to disparate model behaviors across demographic groups. While existing studies primarily focus on detecting and quantifying such biases, they offer limited insight into the underlying mechanisms within the models. To address this gap, we propose an explanatory framework that combines information flow analysis with multi-round dialogue evaluation, aiming to understand the origin of social bias from the perspective of imbalanced internal information utilization. Specifically, we first identify high-contribution image tokens involved in the model's reasoning process for neutral questions via information flow analysis. Then, we design a multi-turn dialogue mechanism to evaluate the extent to which these key tokens encode sensitive information. Extensive experiments reveal that LVLMs exhibit systematic disparities in information usage when processing images of different demographic groups, suggesting that social bias is deeply rooted in the model's internal reasoning dynamics. Furthermore, we complement our findings from a textual modality perspective, showing that the model's semantic representations already display biased proximity patterns, thereby offering a cross-modal explanation of bias formation.