Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition

TL;DR

This paper introduces a novel AVSR method that leverages cross-modal attention to enhance video features by modeling temporal dynamics, significantly improving speech recognition accuracy in noisy environments.

Contribution

It proposes a new approach that learns three types of temporal dynamics in video data and uses cross-modal attention to integrate audio information, advancing AVSR robustness.

Findings

Achieves state-of-the-art results on LRS2 and LRS3 benchmarks in noise conditions.

Excels in recognizing speech amidst babble and speech noise.

Validates the effectiveness of temporal dynamics modeling and cross-modal attention through ablation studies.

Abstract

Audio-visual speech recognition (AVSR) aims to transcribe human speech using both audio and video modalities. In practical environments with noise-corrupted audio, the role of video information becomes crucial. However, prior works have primarily focused on enhancing audio features in AVSR, overlooking the importance of video features. In this study, we strengthen the video features by learning three temporal dynamics in video data: context order, playback direction, and the speed of video frames. Cross-modal attention modules are introduced to enrich video features with audio information so that speech variability can be taken into account when training on the video temporal dynamics. Based on our approach, we achieve the state-of-the-art performance on the LRS2 and LRS3 AVSR benchmarks for the noise-dominant settings. Our approach excels in scenarios especially for babble and speech noise, indicating the ability to distinguish the speech signal that should be recognized from lip movements in the video modality. We support the validity of our methodology by offering the ablation experiments for the temporal dynamics losses and the cross-modal attention architecture design.