mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition

TL;DR

This paper introduces mWhisper-Flamingo, a multilingual audio-visual speech recognition model that combines pre-trained audio and video models, improving noise robustness and achieving state-of-the-art results across nine languages.

Contribution

It presents a novel multimodal integration method with decoder modality dropout, enhancing multilingual AVSR performance in noisy environments.

Findings

Achieves state-of-the-art WER on MuAViC dataset.

Outperforms audio-only Whisper in noisy conditions across all tested languages.

Demonstrates effective multilingual audio-visual speech recognition.

Abstract

Audio-Visual Speech Recognition (AVSR) combines lip-based video with audio and can improve performance in noise, but most methods are trained only on English data. One limitation is the lack of large-scale multilingual video data, which makes it hard to train models from scratch. In this work, we propose mWhisper-Flamingo for multilingual AVSR which combines the strengths of a pre-trained audio model (Whisper) and video model (AV-HuBERT). To enable better multi-modal integration and improve the noisy multilingual performance, we introduce decoder modality dropout where the model is trained both on paired audio-visual inputs and separate audio/visual inputs. mWhisper-Flamingo achieves state-of-the-art WER on MuAViC, an AVSR dataset of 9 languages. Audio-visual mWhisper-Flamingo consistently outperforms audio-only Whisper on all languages in noisy conditions.