From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship

TL;DR

This paper introduces Genres, a new benchmark to evaluate gender bias in multimodal large language models by analyzing social relationship narratives, revealing persistent biases in interactions that previous benchmarks overlooked.

Contribution

The paper presents Genres, a novel benchmark for assessing relational gender bias in MLLMs through social relationship narratives, highlighting subtle biases in interpersonal contexts.

Findings

MLLMs exhibit persistent, context-sensitive gender biases.

Biases are more evident in relational interactions than in isolated scenarios.

The benchmark reveals subtle, interaction-driven gender biases in both open- and closed-source models.

Abstract

Multimodal large language models (MLLMs) have shown impressive capabilities across tasks involving both visual and textual modalities. However, growing concerns remain about their potential to encode and amplify gender bias, particularly in socially sensitive applications. Existing benchmarks predominantly evaluate bias in isolated scenarios, overlooking how bias may emerge subtly through interpersonal interactions. We fill this gap by going beyond single-entity evaluation and instead focusing on a deeper examination of relational and contextual gender bias in dual-individual interactions. We introduce Genres, a novel benchmark designed to evaluate gender bias in MLLMs through the lens of social relationships in generated narratives. Genres assesses gender bias through a dual-character profile and narrative generation task that captures rich interpersonal dynamics and supports a fine-grained bias evaluation suite across multiple dimensions. Experiments on both open- and closed-source MLLMs reveal persistent, context-sensitive gender biases that are not evident in single-character settings. Our findings underscore the importance of relationship-aware benchmarks for diagnosing subtle, interaction-driven gender bias in MLLMs and provide actionable insights for future bias mitigation.