Purification Before Fusion: Toward Mask-Free Speech Enhancement for Robust Audio-Visual Speech Recognition

TL;DR

This paper introduces a mask-free, end-to-end audio-visual speech recognition framework that enhances robustness in noisy environments by implicitly refining audio features with video assistance, outperforming mask-based methods.

Contribution

The proposed framework eliminates the need for explicit noise masks and improves noise robustness by leveraging a Conformer-based fusion module for implicit audio feature refinement.

Findings

Outperforms mask-based baselines on LRS3 benchmark in noisy conditions.

Effectively preserves speech semantics while reducing noise interference.

Demonstrates robustness without explicit noise masking strategies.

Abstract

Audio-visual speech recognition (AVSR) typically improves recognition accuracy in noisy environments by integrating noise-immune visual cues with audio signals. Nevertheless, high-noise audio inputs are prone to introducing adverse interference into the feature fusion process. To mitigate this, recent AVSR methods often adopt mask-based strategies to filter audio noise during feature interaction and fusion, yet such methods risk discarding semantically relevant information alongside noise. In this work, we propose an end-to-end noise-robust AVSR framework coupled with speech enhancement, eliminating the need for explicit noise mask generation. This framework leverages a Conformer-based bottleneck fusion module to implicitly refine noisy audio features with video assistance. By reducing modality redundancy and enhancing inter-modal interactions, our method preserves speech semantic integrity to achieve robust recognition performance. Experimental evaluations on the public LRS3 benchmark suggest that our method outperforms prior advanced mask-based baselines under noisy conditions.