The Voice Behind the Words: Quantifying Intersectional Bias in SpeechLLMs

TL;DR

This study evaluates intersectional accent and gender biases in SpeechLLMs, revealing consistent disparities in helpfulness scores that are more pronounced with human evaluators, highlighting implicit biases in speech processing models.

Contribution

It provides the first large-scale intersectional bias assessment in SpeechLLMs using controlled voice cloning and multiple evaluation methods.

Findings

Eastern European accents receive lower helpfulness scores.

Bias is implicit, affecting helpfulness but not politeness.

Humans detect sharper disparities than LLM judges.

Abstract

Speech Large Language Models (SpeechLLMs) process spoken input directly, retaining cues such as accent and perceived gender that were previously removed in cascaded pipelines. This introduces speaker identity dependent variation in responses. We present a large-scale intersectional evaluation of accent and gender bias in three SpeechLLMs using 2,880 controlled interactions across six English accents and two gender presentations, keeping linguistic content constant through voice cloning. Using pointwise LLM-judge ratings, pairwise comparisons, and Best-Worst Scaling with human validation, we detect consistent disparities. Eastern European-accented speech receives lower helpfulness scores, particularly for female-presenting voices. The bias is implicit: responses remain polite but differ in helpfulness. While LLM judges capture the directional trend of these biases, human evaluators exhibit significantly higher sensitivity, uncovering sharper intersectional disparities.