CTC-DRO: Robust Optimization for Reducing Language Disparities in Speech Recognition

TL;DR

This paper introduces CTC-DRO, a robust optimization method designed to improve multilingual speech recognition by reducing language disparities and addressing limitations of existing group DRO approaches.

Contribution

The paper proposes CTC-DRO, a novel method that smooths group weight updates and uses input length-matched batching to enhance robustness in speech recognition models.

Findings

CTC-DRO reduces worst-language error by up to 47.1%.

It outperforms group DRO and baseline models in multilingual ASR.

Minimal additional computational costs are required.

Abstract

Modern deep learning models often achieve high overall performance, but consistently fail on specific subgroups. Group distributionally robust optimization (group DRO) addresses this problem by minimizing the worst-group loss, but it fails when group losses misrepresent performance differences between groups. This is common in domains like speech, where the widely used connectionist temporal classification (CTC) loss not only scales with input length but also varies with linguistic and acoustic properties, leading to spurious differences between group losses. We present CTC-DRO, which addresses the shortcomings of the group DRO objective by smoothing the group weight update to prevent overemphasis on consistently high-loss groups, while using input length-matched batching to mitigate CTC's scaling issues. We evaluate CTC-DRO on the task of multilingual automatic speech recognition (ASR) across five language sets from the diverse ML-SUPERB 2.0 benchmark. CTC-DRO consistently outperforms group DRO and CTC-based baseline models, reducing the worst-language error by up to 47.1% and the average error by up to 32.9%. CTC-DRO can be applied to ASR with minimal computational costs, and, while motivated by multilingual ASR, offers the potential for reducing group disparities in other domains with similar challenges.