Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generation

TL;DR

This paper enhances bilingual neural transducer ASR models for code-switching speech by leveraging synthetic data and semi-supervised training, achieving significant error rate reductions without requiring supervised code-switching data.

Contribution

It introduces a semi-supervised training approach with synthetic code-switching data to improve bilingual ASR performance in code-switching scenarios.

Findings

Achieved 25% MER on ASCEND dataset, reducing previous errors by 2.1%.

Synthetic data and semi-supervised training improve code-switching recognition.

Maintains high accuracy on monolingual speech.

Abstract

Code-switching describes the practice of using more than one language in the same sentence. In this study, we investigate how to optimize a neural transducer based bilingual automatic speech recognition (ASR) model for code-switching speech. Focusing on the scenario where the ASR model is trained without supervised code-switching data, we found that semi-supervised training and synthetic code-switched data can improve the bilingual ASR system on code-switching speech. We analyze how each of the neural transducer's encoders contributes towards code-switching performance by measuring encoder-specific recall values, and evaluate our English/Mandarin system on the ASCEND data set. Our final system achieves 25% mixed error rate (MER) on the ASCEND English/Mandarin code-switching test set -- reducing the MER by 2.1% absolute compared to the previous literature -- while maintaining good accuracy on the monolingual test sets.