Adaptive multilingual speech recognition with pretrained models

TL;DR

This paper explores leveraging pretrained audio and text models with adaptive weighting to enhance multilingual speech recognition, achieving significant improvements especially in low-resource languages.

Contribution

It introduces a novel approach combining wav2vec 2.0 and MBART50 with adaptive weights, demonstrating substantial performance gains over traditional supervised methods.

Findings

44% improvement over supervised learning

Different techniques reinforce different languages

Potential for further enhancement by architectural modifications

Abstract

Multilingual speech recognition with supervised learning has achieved great results as reflected in recent research. With the development of pretraining methods on audio and text data, it is imperative to transfer the knowledge from unsupervised multilingual models to facilitate recognition, especially in many languages with limited data. Our work investigated the effectiveness of using two pretrained models for two modalities: wav2vec 2.0 for audio and MBART50 for text, together with the adaptive weight techniques to massively improve the recognition quality on the public datasets containing CommonVoice and Europarl. Overall, we noticed an 44% improvement over purely supervised learning, and more importantly, each technique provides a different reinforcement in different languages. We also explore other possibilities to potentially obtain the best model by slightly adding either depth or relative attention to the architecture.