Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment

TL;DR

This paper introduces a novel approach to code-switch speech translation by enhancing Large Language Models with a Mixture-of-Experts speech projector, improving semantic modeling and translation accuracy in multilingual scenarios.

Contribution

It proposes a Mixture-of-Experts speech projector with language-specific training and a multi-stage paradigm to improve code-switch speech translation performance.

Findings

Achieved an average of 0.86 BLEU and 0.93 COMET improvements over SeamlessM4T.

Demonstrated effectiveness across multiple datasets and test sets.

Enhanced semantic space alignment improves translation in code-switch scenarios.

Abstract

Code-switching (CS) speech translation (ST) aims to translate speech that alternates between multiple languages into a target language text, posing significant challenges due to the complexity of semantic modeling and the scarcity of CS data. Previous studies mainly rely on the models themselves to implicitly learn semantic representations and resort to costly manual annotations. To mitigate these limitations, we propose enhancing Large Language Models (LLMs) with a Mixture-of-Experts (MoE) speech projector composed of language expert groups, where each group specializes in the semantic space of a specific language for fine-grained speech feature modeling. A language-specific loss and an intra-group load balancing loss are jointly introduced to guide efficient token routing across and within expert groups. Furthermore, we introduce a multi-stage training paradigm that utilizes readily available automatic speech recognition (ASR) and monolingual ST data, facilitating speech-text alignment and improving translation performance. To bridge the data gap for smooth domain transfer, a transition loss is employed to improve adaptation to CS scenarios. Extensive experiments on widely used datasets demonstrate the effectiveness and generality of our approach, achieving average improvements of $0.86$ BLEU and $0.93$ COMET over SeamlessM4T, with maximum improvements of $1.49$ BLEU and $1.41$ COMET across different test sets.