Predict-and-Update Network: Audio-Visual Speech Recognition Inspired by Human Speech Perception

TL;DR

This paper introduces a Predict-and-Update Network that mimics human visual cueing to improve audio-visual speech recognition, significantly reducing word error rates especially in noisy environments.

Contribution

It proposes a novel visual cueing mechanism with a cross-modal Conformer, enhancing AVSR performance beyond existing methods.

Findings

Outperforms state-of-the-art AVSR methods on LRS2-BBC and LRS3-BBC datasets.

Reduces Word Error Rate by over 10% in clean and 40% in noisy conditions.

Validates the effectiveness of visual cueing in multi-modal speech recognition.

Abstract

Audio and visual signals complement each other in human speech perception, so do they in speech recognition. The visual hint is less evident than the acoustic hint, but more robust in a complex acoustic environment, as far as speech perception is concerned. It remains a challenge how we effectively exploit the interaction between audio and visual signals for automatic speech recognition. There have been studies to exploit visual signals as redundant or complementary information to audio input in a synchronous manner. Human studies suggest that visual signal primes the listener in advance as to when and on which frequency to attend to. We propose a Predict-and-Update Network (P&U net), to simulate such a visual cueing mechanism for Audio-Visual Speech Recognition (AVSR). In particular, we first predict the character posteriors of the spoken words, i.e. the visual embedding, based on the visual signals. The audio signal is then conditioned on the visual embedding via a novel cross-modal Conformer, that updates the character posteriors. We validate the effectiveness of the visual cueing mechanism through extensive experiments. The proposed P&U net outperforms the state-of-the-art AVSR methods on both LRS2-BBC and LRS3-BBC datasets, with the relative reduced Word Error Rate (WER)s exceeding 10% and 40% under clean and noisy conditions, respectively.