Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities

TL;DR

This paper investigates performance disparities in speech recognition systems across different user groups, proposing methods to discover and mitigate these disparities to improve fairness without sacrificing accuracy.

Contribution

It introduces scalable cohort discovery methods using speaker embeddings and evaluates mitigation strategies like oversampling and additional input variables.

Findings

Speaker embeddings effectively identify diverse cohorts.

Oversampling reduces performance gaps.

Fairness mitigation does not harm overall accuracy.

Abstract

As for other forms of AI, speech recognition has recently been examined with respect to performance disparities across different user cohorts. One approach to achieve fairness in speech recognition is to (1) identify speaker cohorts that suffer from subpar performance and (2) apply fairness mitigation measures targeting the cohorts discovered. In this paper, we report on initial findings with both discovery and mitigation of performance disparities using data from a product-scale AI assistant speech recognition system. We compare cohort discovery based on geographic and demographic information to a more scalable method that groups speakers without human labels, using speaker embedding technology. For fairness mitigation, we find that oversampling of underrepresented cohorts, as well as modeling speaker cohort membership by additional input variables, reduces the gap between top- and bottom-performing cohorts, without deteriorating overall recognition accuracy.