ASR-FAIRBENCH: Measuring and Benchmarking Equity Across Speech Recognition Systems

TL;DR

This paper introduces ASR-FAIRBENCH, a comprehensive benchmarking framework for evaluating both accuracy and fairness of speech recognition systems across diverse demographic groups, highlighting disparities and guiding more inclusive development.

Contribution

We present the ASR-FAIRBENCH leaderboard and a novel fairness score based on demographic data, enabling real-time assessment of ASR systems' equity and accuracy.

Findings

Significant performance disparities across demographic groups in SOTA ASR models

The FAIRBENCH framework effectively measures both accuracy and fairness

Benchmark results highlight the need for more inclusive ASR development

Abstract

Automatic Speech Recognition (ASR) systems have become ubiquitous in everyday applications, yet significant disparities in performance across diverse demographic groups persist. In this work, we introduce the ASR-FAIRBENCH leaderboard which is designed to assess both the accuracy and equity of ASR models in real-time. Leveraging the Meta's Fair-Speech dataset, which captures diverse demographic characteristics, we employ a mixed-effects Poisson regression model to derive an overall fairness score. This score is integrated with traditional metrics like Word Error Rate (WER) to compute the Fairness Adjusted ASR Score (FAAS), providing a comprehensive evaluation framework. Our approach reveals significant performance disparities in SOTA ASR models across demographic groups and offers a benchmark to drive the development of more inclusive ASR technologies.