Multilingual Speech Translation with Efficient Finetuning of Pretrained Models

TL;DR

This paper introduces a minimalistic finetuning method for multilingual speech translation that leverages pretrained models, achieving state-of-the-art results with low training cost and strong zero-shot capabilities.

Contribution

It demonstrates that finetuning less than 10% of pretrained parameters with LayerNorm and Attention enables effective multilingual speech translation and zero-shot transfer.

Findings

Achieved +6.4 BLEU on average across 15 En-X directions

Surpassed cascaded ST in 23 out of 34 directions

Demonstrated strong zero-shot performance with +5.7 BLEU on average

Abstract

We present a simple yet effective approach to build multilingual speech-to-text (ST) translation by efficient transfer learning from pretrained speech encoder and text decoder. Our key finding is that a minimalistic LNA (LayerNorm and Attention) finetuning can achieve zero-shot crosslingual and cross-modality transfer ability by only finetuning less than 10% of the pretrained parameters. This enables effectively leveraging large pretrained models with low training cost. Using wav2vec 2.0 for acoustic modeling, and mBART for multilingual text generation, our approach advanced the new state-of-the-art for 34 translation directions (and surpassing cascaded ST for 23 of them) on large-scale multilingual ST benchmark CoVoST 2 (+6.4 BLEU on average across 15 En-X directions and +5.1 BLEU on average across 19 X-En directions). Our approach demonstrates strong zero-shot performance in a many-to-many multilingual model (+5.7 BLEU on average across 18 non-English directions), making it an appealing approach for attaining high-quality speech translation with improved parameter and data efficiency.