Audio-Visual Speech Enhancement In Complex Scenarios With Separation And Dereverberation Joint Modeling

TL;DR

This paper introduces a novel audio-visual speech enhancement system designed for complex acoustic environments, employing a separation before dereverberation pipeline, which outperforms previous methods and achieves top results in a competitive challenge.

Contribution

The paper proposes a new AVSE approach with a separation-then-dereverberation pipeline, enhancing speech quality in complex scenarios and extending applicability to other AVSE networks.

Findings

Achieved top performance in objective metrics on AVSEC-4

Secured first place in human subjective listening test

Demonstrated robustness in complex acoustic environments

Abstract

Audio-visual speech enhancement (AVSE) is a task that uses visual auxiliary information to extract a target speaker's speech from mixed audio. In real-world scenarios, there often exist complex acoustic environments, accompanied by various interfering sounds and reverberation. Most previous methods struggle to cope with such complex conditions, resulting in poor perceptual quality of the extracted speech. In this paper, we propose an effective AVSE system that performs well in complex acoustic environments. Specifically, we design a "separation before dereverberation" pipeline that can be extended to other AVSE networks. The 4th COGMHEAR Audio-Visual Speech Enhancement Challenge (AVSEC) aims to explore new approaches to speech processing in multimodal complex environments. We validated the performance of our system in AVSEC-4: we achieved excellent results in the three objective metrics on the competition leaderboard, and ultimately secured first place in the human subjective listening test.