Vision-Language Models display a strong gender bias

TL;DR

This paper investigates gender bias in vision-language models by analyzing how these models associate face images with occupation and activity descriptions, revealing subtle gender stereotypes encoded in their representations.

Contribution

It introduces a novel framework for measuring and visualizing gender bias in contrastive vision-language models using a comprehensive dataset and statistical analysis methods.

Findings

Models show significant gender-linked associations with certain occupations.

Gender bias varies across different categories of activities and roles.

The framework provides a robust way to quantify and interpret bias in VLMs.

Abstract

Vision-language models (VLM) align images and text in a shared representation space that is useful for retrieval and zero-shot transfer. Yet, this alignment can encode and amplify social stereotypes in subtle ways that are not obvious from standard accuracy metrics. In this study, we test whether the contrastive vision-language encoder exhibits gender-linked associations when it places embeddings of face images near embeddings of short phrases that describe occupations and activities. We assemble a dataset of 220 face photographs split by perceived binary gender and a set of 150 unique statements distributed across six categories covering emotional labor, cognitive labor, domestic labor, technical labor, professional roles, and physical labor. We compute unit-norm image embeddings for every face and unit-norm text embeddings for every statement, then define a statement-level association score as the difference between the mean cosine similarity to the male set and the mean cosine similarity to the female set, where positive values indicate stronger association with the male set and negative values indicate stronger association with the female set. We attach bootstrap confidence intervals by resampling images within each gender group, aggregate by category with a separate bootstrap over statements, and run a label-swap null model that estimates the level of mean absolute association we would expect if no gender structure were present. The outcome is a statement-wise and category-wise map of gender associations in a contrastive vision-language space, accompanied by uncertainty, simple sanity checks, and a robust gender bias evaluation framework.