Monolingual Recognizers Fusion for Code-switching Speech Recognition

TL;DR

This paper introduces a novel monolingual recognizers fusion approach for code-switching speech recognition, utilizing a two-stage process to improve performance by effectively combining language-specific predictions.

Contribution

It proposes a two-stage fusion method with language-aware training and a text simulation strategy, enabling better use of pre-trained monolingual models for code-switching ASR.

Findings

Significant reduction in mix error rate on Mandarin-English corpus

Effective fusion of monolingual models improves code-switching recognition

Proposed method outperforms existing bi-encoder approaches

Abstract

The bi-encoder structure has been intensively investigated in code-switching (CS) automatic speech recognition (ASR). However, most existing methods require the structures of two monolingual ASR models (MAMs) should be the same and only use the encoder of MAMs. This leads to the problem that pre-trained MAMs cannot be timely and fully used for CS ASR. In this paper, we propose a monolingual recognizers fusion method for CS ASR. It has two stages: the speech awareness (SA) stage and the language fusion (LF) stage. In the SA stage, acoustic features are mapped to two language-specific predictions by two independent MAMs. To keep the MAMs focused on their own language, we further extend the language-aware training strategy for the MAMs. In the LF stage, the BELM fuses two language-specific predictions to get the final prediction. Moreover, we propose a text simulation strategy to simplify the training process of the BELM and reduce reliance on CS data. Experiments on a Mandarin-English corpus show the efficiency of the proposed method. The mix error rate is significantly reduced on the test set after using open-source pre-trained MAMs.