Refusal as Silence: Gendered Disparities in Vision-Language Model Responses

TL;DR

This paper explores how gender identity influences refusal behavior in vision-language models, revealing disparities that impact fairness and access, and highlighting methodological considerations for equity in AI systems.

Contribution

It introduces a counterfactual persona approach to analyze gendered disparities in model refusals, advancing understanding of bias in AI content moderation.

Findings

Transgender and non-binary personas face higher refusal rates.

Refusal disparities persist even in non-harmful contexts.

Methodological insights for equity audits using LLMs.

Abstract

Refusal behavior by Large Language Models is increasingly visible in content moderation, yet little is known about how refusals vary by the identity of the user making the request. This study investigates refusal as a sociotechnical outcome through a counterfactual persona design that varies gender identity--including male, female, non-binary, and transgender personas--while keeping the classification task and visual input constant. Focusing on a vision-language model (GPT-4V), we examine how identity-based language cues influence refusal in binary gender classification tasks. We find that transgender and non-binary personas experience significantly higher refusal rates, even in non-harmful contexts. Our findings also provide methodological implications for equity audits and content analysis using LLMs. Our findings underscore the importance of modeling identity-driven disparities and caution against uncritical use of AI systems for content coding. This study advances algorithmic fairness by reframing refusal as a communicative act that may unevenly regulate epistemic access and participation.