Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

TL;DR

This paper introduces AV-HuBERT, a self-supervised framework that learns audio-visual speech representations by predicting masked multimodal hidden units, significantly improving lip-reading and speech recognition performance with less labeled data.

Contribution

The paper presents AV-HuBERT, a novel self-supervised learning method that leverages masked multimodal inputs and predicted hidden units for superior audio-visual speech representation.

Findings

Achieves 32.5% WER on LRS3 with only 30 hours of labeled data.

Outperforms previous state-of-the-art with much less labeled data.

Reduces WER to 26.9% with full labeled dataset and self-training.

Abstract

Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/facebookresearch/av_hubert