Enhancing Code-Switching ASR Leveraging Non-Peaky CTC Loss and Deep Language Posterior Injection

TL;DR

This paper introduces a novel approach to improve code-switching automatic speech recognition by integrating language identification, boundary alignment loss, and deep language posterior interaction, leading to better language handling in E2E ASR systems.

Contribution

It proposes a new method combining LID embedding, boundary alignment loss, and deep language posterior interaction to enhance code-switching ASR performance.

Findings

Outperforms prior D-MoE method on SEAME corpus

Enriches encoder with detailed language information

Improves language handling in end-to-end ASR systems

Abstract

Code-switching-where multilingual speakers alternately switch between languages during conversations-still poses significant challenges to end-to-end (E2E) automatic speech recognition (ASR) systems due to phenomena of both acoustic and semantic confusion. This issue arises because ASR systems struggle to handle the rapid alternation of languages effectively, which often leads to significant performance degradation. Our main contributions are at least threefold: First, we incorporate language identification (LID) information into several intermediate layers of the encoder, aiming to enrich output embeddings with more detailed language information. Secondly, through the novel application of language boundary alignment loss, the subsequent ASR modules are enabled to more effectively utilize the knowledge of internal language posteriors. Third, we explore the feasibility of using language posteriors to facilitate deep interaction between shared encoder and language-specific encoders. Through comprehensive experiments on the SEAME corpus, we have verified that our proposed method outperforms the prior-art method, disentangle based mixture-of-experts (D-MoE), further enhancing the acuity of the encoder to languages.