Inference and Denoise: Causal Inference-based Neural Speech Enhancement

TL;DR

This paper introduces a causal inference framework for speech enhancement that models noise as an intervention, leading to improved performance and efficiency over traditional methods.

Contribution

It proposes a novel causal inference-based speech enhancement method that uses noise as an intervention and a noise detector to improve denoising accuracy.

Findings

CISE outperforms non-causal mask-based SE methods.

CISE achieves better performance and efficiency than complex SE models.

A SE-specific average treatment effect is derived for causal quantification.

Abstract

This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention. Based on the potential outcome framework, the proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE. Specifically, we use the presence of noise as guidance for EM selection during training, and the noise detector selects the enhancement module according to the prediction of the presence of noise for each frame. Moreover, we derived a SE-specific average treatment effect to quantify the causal effect adequately. Experimental evidence demonstrates that CISE outperforms a non-causal mask-based SE approach in the studied settings and has better performance and efficiency than more complex SE models.