Noise-Aware Speech Separation with Contrastive Learning

TL;DR

This paper introduces a noise-aware speech separation method that uses contrastive learning to improve the quality of separated speech signals in noisy environments, achieving significant performance gains.

Contribution

It proposes a novel noise-aware framework with patch-wise contrastive learning to effectively suppress background noise in speech separation tasks.

Findings

Achieves 1-2dB SI-SNRi or SDRi improvements over DPRNN and Sepformer.

Effectively suppresses background noise with minimal parameter increase.

Demonstrates robustness on WHAM! and LibriMix noisy datasets.

Abstract

Recently, speech separation (SS) task has achieved remarkable progress driven by deep learning technique. However, it is still challenging to separate target speech from noisy mixture, as the neural model is vulnerable to assign background noise to each speaker. In this paper, we propose a noise-aware SS (NASS) method, which aims to improve the speech quality for separated signals under noisy conditions. Specifically, NASS views background noise as an additional output and predicts it along with other speakers in a mask-based manner. To effectively denoise, we introduce patch-wise contrastive learning (PCL) between noise and speaker representations from the decoder input and encoder output. PCL loss aims to minimize the mutual information between predicted noise and other speakers at multiple-patch level to suppress the noise information in separated signals. Experimental results show that NASS achieves 1 to 2dB SI-SNRi or SDRi over DPRNN and Sepformer on WHAM! and LibriMix noisy datasets, with less than 0.1M parameter increase.