Modality Attention for End-to-End Audio-visual Speech Recognition

TL;DR

This paper introduces a novel modality attention mechanism for end-to-end audio-visual speech recognition, significantly improving robustness and accuracy in noisy environments by adaptively fusing audio and visual data.

Contribution

It proposes a state-of-the-art multimodal attention approach within Seq2seq architectures for AVSR, outperforming traditional feature concatenation methods.

Findings

Achieves up to 36% improvement over audio-only systems in noisy conditions.

Outperforms traditional feature concatenation in both clean and noisy environments.

Demonstrates the generalizability of modality attention to other multimodal tasks.

Abstract

Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for robust speech recognition, especially in noisy environment. In this paper, we propose a novel multimodal attention based method for audio-visual speech recognition which could automatically learn the fused representation from both modalities based on their importance. Our method is realized using state-of-the-art sequence-to-sequence (Seq2seq) architectures. Experimental results show that relative improvements from 2% up to 36% over the auditory modality alone are obtained depending on the different signal-to-noise-ratio (SNR). Compared to the traditional feature concatenation methods, our proposed approach can achieve better recognition performance under both clean and noisy conditions. We believe modality attention based end-to-end method can be easily generalized to other multimodal tasks with correlated information.