Who Gets the Mic? Investigating Gender Bias in the Speaker Assignment of a Speech-LLM

TL;DR

This paper investigates gender bias in Speech-LLMs by analyzing speaker assignment patterns in Bark, revealing gender awareness and inclinations but no systematic bias, using specially constructed datasets.

Contribution

It introduces a novel methodology using speaker assignment as an explicit bias analysis tool for Speech-LLMs, expanding bias detection beyond text-based models.

Findings

Bark shows gender awareness and some inclinations.

No systematic gender bias found in speaker assignment.

Constructed datasets effectively reveal gender-related patterns.

Abstract

Similar to text-based Large Language Models (LLMs), Speech-LLMs exhibit emergent abilities and context awareness. However, whether these similarities extend to gender bias remains an open question. This study proposes a methodology leveraging speaker assignment as an analytic tool for bias investigation. Unlike text-based models, which encode gendered associations implicitly, Speech-LLMs must produce a gendered voice, making speaker selection an explicit bias cue. We evaluate Bark, a Text-to-Speech (TTS) model, analyzing its default speaker assignments for textual prompts. If Bark's speaker selection systematically aligns with gendered associations, it may reveal patterns in its training data or model design. To test this, we construct two datasets: (i) Professions, containing gender-stereotyped occupations, and (ii) Gender-Colored Words, featuring gendered connotations. While Bark does not exhibit systematic bias, it demonstrates gender awareness and has some gender inclinations.