CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition

TL;DR

CAMEL introduces a novel cross-attention-based method to enhance language-specific speech representations and incorporate language bias, significantly improving code-switching speech recognition accuracy across multiple datasets.

Contribution

The paper proposes CAMEL, a cross-attention enhanced MoE and language bias approach, advancing beyond simple fusion methods for better code-switching ASR performance.

Findings

Achieves state-of-the-art results on SEAME, ASRU200, and ASRU700+LibriSpeech460 datasets.

Effectively models language-specific speech representations with cross-attention.

Incorporates language bias from the LD decoder to improve transcription accuracy.

Abstract

Code-switching automatic speech recognition (ASR) aims to transcribe speech that contains two or more languages accurately. To better capture language-specific speech representations and address language confusion in code-switching ASR, the mixture-of-experts (MoE) architecture and an additional language diarization (LD) decoder are commonly employed. However, most researches remain stagnant in simple operations like weighted summation or concatenation to fuse languagespecific speech representations, leaving significant opportunities to explore the enhancement of integrating language bias information. In this paper, we introduce CAMEL, a cross-attention-based MoE and language bias approach for code-switching ASR. Specifically, after each MoE layer, we fuse language-specific speech representations with cross-attention, leveraging its strong contextual modeling abilities. Additionally, we design a source attention-based mechanism to incorporate the language information from the LD decoder output into text embeddings. Experimental results demonstrate that our approach achieves state-of-the-art performance on the SEAME, ASRU200, and ASRU700+LibriSpeech460 Mandarin-English code-switching ASR datasets.