LAE-ST-MoE: Boosted Language-Aware Encoder Using Speech Translation Auxiliary Task for E2E Code-switching ASR

TL;DR

This paper introduces LAE-ST-MoE, a novel framework that enhances code-switching ASR by integrating speech translation tasks to leverage contextual language information, resulting in improved accuracy.

Contribution

The paper proposes a new LAE-ST-MoE model that incorporates speech translation tasks into language-aware encoders using a mixture of experts, improving code-switching ASR performance.

Findings

Achieves 9.26% reduction in mix error on CS test dataset.

Enables speech translation from CS speech to Mandarin or English text.

Demonstrates effectiveness of integrating translation tasks into ASR models.

Abstract

Recently, to mitigate the confusion between different languages in code-switching (CS) automatic speech recognition (ASR), the conditionally factorized models, such as the language-aware encoder (LAE), explicitly disregard the contextual information between different languages. However, this information may be helpful for ASR modeling. To alleviate this issue, we propose the LAE-ST-MoE framework. It incorporates speech translation (ST) tasks into LAE and utilizes ST to learn the contextual information between different languages. It introduces a task-based mixture of expert modules, employing separate feed-forward networks for the ASR and ST tasks. Experimental results on the ASRU 2019 Mandarin-English CS challenge dataset demonstrate that, compared to the LAE-based CTC, the LAE-ST-MoE model achieves a 9.26% mix error reduction on the CS test with the same decoding parameter. Moreover, the well-trained LAE-ST-MoE model can perform ST tasks from CS speech to Mandarin or English text.