mSLAM: Massively multilingual joint pre-training for speech and text

TL;DR

mSLAM is a large-scale multilingual model that jointly pre-trains on speech and text, enabling cross-modal understanding and zero-shot translation, with improved performance on various speech tasks.

Contribution

This work introduces mSLAM, a novel joint pre-training framework for speech and text that enhances cross-lingual and cross-modal representations in a single model.

Findings

Improves speech translation, intent classification, and language identification.

Achieves zero-shot text translation without explicit text translation data.

Benefits from multi-modal fine-tuning to further enhance speech translation quality.

Abstract

We present mSLAM, a multilingual Speech and LAnguage Model that learns cross-lingual cross-modal representations of speech and text by pre-training jointly on large amounts of unlabeled speech and text in multiple languages. mSLAM combines w2v-BERT pre-training on speech with SpanBERT pre-training on character-level text, along with Connectionist Temporal Classification (CTC) losses on paired speech and transcript data, to learn a single model capable of learning from and representing both speech and text signals in a shared representation space. We evaluate mSLAM on several downstream speech understanding tasks and find that joint pre-training with text improves quality on speech translation, speech intent classification and speech language-ID while being competitive on multilingual ASR, when compared against speech-only pre-training. Our speech translation model demonstrates zero-shot text translation without seeing any text translation data, providing evidence for cross-modal alignment of representations. mSLAM also benefits from multi-modal fine-tuning, further improving the quality of speech translation by directly leveraging text translation data during the fine-tuning process. Our empirical analysis highlights several opportunities and challenges arising from large-scale multimodal pre-training, suggesting directions for future research.