AVATAR: Unconstrained Audiovisual Speech Recognition

TL;DR

This paper introduces AVATAR, a novel end-to-end audiovisual speech recognition model that leverages full-frame visual information, improving recognition especially in noisy conditions and unconstrained videos.

Contribution

The paper presents AVATAR, a new sequence-to-sequence model that incorporates full-frame visual cues and novel training strategies for enhanced audiovisual speech recognition.

Findings

AVATAR outperforms prior models on the How2 benchmark.

Visual modality significantly improves recognition in noisy environments.

Created a new real-world AV-ASR dataset, VisSpeech.

Abstract

Audio-visual automatic speech recognition (AV-ASR) is an extension of ASR that incorporates visual cues, often from the movements of a speaker's mouth. Unlike works that simply focus on the lip motion, we investigate the contribution of entire visual frames (visual actions, objects, background etc.). This is particularly useful for unconstrained videos, where the speaker is not necessarily visible. To solve this task, we propose a new sequence-to-sequence AudioVisual ASR TrAnsformeR (AVATAR) which is trained end-to-end from spectrograms and full-frame RGB. To prevent the audio stream from dominating training, we propose different word-masking strategies, thereby encouraging our model to pay attention to the visual stream. We demonstrate the contribution of the visual modality on the How2 AV-ASR benchmark, especially in the presence of simulated noise, and show that our model outperforms all other prior work by a large margin. Finally, we also create a new, real-world test bed for AV-ASR called VisSpeech, which demonstrates the contribution of the visual modality under challenging audio conditions.