Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement

TL;DR

This paper introduces DLAV-SE, a diffusion-based multi-modal speech enhancement framework that integrates audio, visual, and linguistic data, significantly improving speech quality by effectively bridging modality gaps through cross-modal knowledge transfer.

Contribution

The paper proposes a novel diffusion-based AVSE model that incorporates linguistic knowledge via a pretrained language model and a cross-modal transfer mechanism, which is embedded into the model during training.

Findings

DLAV-SE outperforms state-of-the-art methods in speech quality.

The approach reduces phonetic confusion and artifacts.

Visualization confirms improved output quality.

Abstract

Speech enhancement (SE) aims to improve the quality and intelligibility of speech in noisy environments. Recent studies have shown that incorporating visual cues in audio signal processing can enhance SE performance. Given that human speech communication naturally involves audio, visual, and linguistic modalities, it is reasonable to expect additional improvements by integrating linguistic information. However, effectively bridging these modality gaps, particularly during knowledge transfer remains a significant challenge. In this paper, we propose a novel multi-modal learning framework, termed DLAV-SE, which leverages a diffusion-based model integrating audio, visual, and linguistic information for audio-visual speech enhancement (AVSE). Within this framework, the linguistic modality is modeled using a pretrained language model (PLM), which transfers linguistic knowledge to the audio-visual domain through a cross-modal knowledge transfer (CMKT) mechanism during training. After training, the PLM is no longer required at inference, as its knowledge is embedded into the AVSE model through the CMKT process. We conduct a series of SE experiments to evaluate the effectiveness of our approach. Results show that the proposed DLAV-SE system significantly improves speech quality and reduces generative artifacts, such as phonetic confusion, compared to state-of-the-art (SOTA) methods. Furthermore, visualization analyses confirm that the CMKT method enhances the generation quality of the AVSE outputs. These findings highlight both the promise of diffusion-based methods for advancing AVSE and the value of incorporating linguistic information to further improve system performance.