Visual-Informed Speech Enhancement Using Attention-Based Beamforming

TL;DR

This paper introduces a novel audiovisual neural beamforming system that uses visual cues and attention mechanisms to improve speech enhancement, especially in challenging acoustic environments with static or moving speakers.

Contribution

It presents a new multimodal neural network that integrates visual speech features with microphone array processing for robust speech enhancement.

Findings

Improved speech enhancement performance over baseline methods.

Enhanced robustness in dynamic speaker scenarios.

Effective use of visual cues for voice activity detection.

Abstract

Recent studies have demonstrated that incorporating auxiliary information, such as speaker voiceprint or visual cues, can substantially improve Speech Enhancement (SE) performance. However, single-channel methods often yield suboptimal results in low signal-to-noise ratio (SNR) conditions, when there is high reverberation, or in complex scenarios involving dynamic speakers, overlapping speech, or non-stationary noise. To address these issues, we propose a novel Visual-Informed Neural Beamforming Network (VI-NBFNet), which integrates microphone array signal processing and deep neural networks (DNNs) using multimodal input features. The proposed network leverages a pretrained visual speech recognition model to extract lip movements as input features, which serve for voice activity detection (VAD) and target speaker identification. The system is intended to handle both static and moving speakers by introducing a supervised end-to-end beamforming framework equipped with an attention mechanism. The experimental results demonstrated that the proposed audiovisual system has achieved better SE performance and robustness for both stationary and dynamic speaker scenarios, compared to several baseline methods.