Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLM

TL;DR

This paper presents a novel speech-conditioned LLM with a Mixture of Experts architecture and an IDIT mechanism to improve code-switching ASR performance, achieving significant improvements over existing models.

Contribution

It introduces a MoE-based connector and IDIT mechanism for better handling of code-switching in speech recognition, along with a two-stage training strategy for enhanced performance.

Findings

Outperforms state-of-the-art models in code-switching ASR tasks

Effective integration of MoE architecture with LLM for multilingual speech recognition

Two-stage training strategy improves model adaptability and accuracy

Abstract

In this paper, we introduce a speech-conditioned Large Language Model (LLM) integrated with a Mixture of Experts (MoE) based connector to address the challenge of Code-Switching (CS) in Automatic Speech Recognition (ASR). Specifically, we propose an Insertion and Deletion of Interruption Token (IDIT) mechanism for better transfer text generation ability of LLM to speech recognition task. We also present a connecter with MoE architecture that manages multiple languages efficiently. To further enhance the collaboration of multiple experts and leverage the understanding capabilities of LLM, we propose a two-stage progressive training strategy: 1) The connector is unfrozen and trained with language-specialized experts to map speech representations to the text space. 2) The connector and LLM LoRA adaptor are trained with the proposed IDIT mechanism and all experts are activated to learn general representations. Experimental results demonstrate that our method significantly outperforms state-of-the-art models, including end-to-end and large-scale audio-language models.