MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages

TL;DR

The paper introduces MCAT, a framework that significantly expands multilingual speech-to-text translation to 70 languages and improves efficiency by reducing speech sequence length, outperforming existing models.

Contribution

It presents a novel multilingual scaling method and an optimized speech adapter to enhance language coverage and inference speed in MLLMs for speech translation.

Findings

Achieves translation among 70 languages with mutual translation capabilities.

Surpasses state-of-the-art models on the FLEURS dataset across 70x69 directions.

Improves inference efficiency by reducing speech sequence length to 30 tokens.

Abstract

Multimodal Large Language Models (MLLMs) have achieved great success in Speech-to-Text Translation (S2TT) tasks. However, current research is constrained by two key challenges: language coverage and efficiency. Most of the popular S2TT datasets are substantially English-centric, which restricts the scaling-up of MLLMs' many-to-many translation capabilities. Moreover, the inference speed of MLLMs degrades dramatically when the speech is converted into long sequences (e.g., 750 tokens). To address these limitations, we propose a Multilingual Cost-effective Accelerated Speech-to-Text Translator (MCAT) framework, which includes two innovations. First, a language scaling method that leverages curriculum learning and a data balancing strategy is introduced to extend the language coverage supported by MLLMs to 70 languages and achieve mutual translation among these languages. Second, an optimized speech adapter module is designed to reduce the length of the speech sequence to only 30 tokens. Extensive experiments were conducted on MLLMs of different scales (9B and 27B). The experimental results demonstrate that MCAT not only surpasses state-of-the-art end-to-end models on the FLEURS dataset across 70x69 directions but also enhances inference efficiency. The code and models are released at https://github.com/yxduir/m2m-70.