MOS-Bias: From Hidden Gender Bias to Gender-Aware Speech Quality Assessment

TL;DR

This paper systematically analyzes gender bias in speech quality assessments, revealing consistent scoring disparities between male and female listeners and proposing a gender-aware model to mitigate this bias.

Contribution

It is the first to analyze gender bias in MOS and introduces a gender-aware model that improves prediction accuracy by learning gender-specific scoring patterns.

Findings

Male listeners assign higher scores than female listeners, especially in low-quality speech.

Gender bias in MOS predictions can be learned and mitigated with a gender-aware model.

Bias diminishes as speech quality improves, but remains systematic and detectable.

Abstract

The Mean Opinion Score (MOS) serves as the standard metric for speech quality assessment, yet biases in human annotations remain underexplored. We conduct the first systematic analysis of gender bias in MOS, revealing that male listeners consistently assign higher scores than female listeners--a gap that is most pronounced in low-quality speech and gradually diminishes as quality improves. This quality-dependent structure proves difficult to eliminate through simple calibration. We further demonstrate that automated MOS models trained on aggregated labels exhibit predictions skewed toward male standards of perception. To address this, we propose a gender-aware model that learns gender-specific scoring patterns through abstracting binary group embeddings, thereby improving overall and gender-specific prediction accuracy. This study establishes that gender bias in MOS constitutes a systematic, learnable pattern demanding attention in equitable speech evaluation.