Reverberation as Supervision for Speech Separation

TL;DR

This paper introduces reverberation as supervision (RAS), a new unsupervised loss for single-channel reverberant speech separation that leverages two-channel mixtures and Wiener filtering to achieve competitive separation performance with limited labeled data.

Contribution

The paper presents RAS, a novel unsupervised loss function for speech separation that does not require mixture synthesis or teacher models, enabling effective semi-supervised training.

Findings

Achieves 70-78% of fully supervised SI-SDR improvement with only 5-10% labeled data.

Outperforms models trained solely on labeled data in semi-supervised settings.

Demonstrates effectiveness of reverberation-based supervision in speech separation.

Abstract

This paper proposes reverberation as supervision (RAS), a novel unsupervised loss function for single-channel reverberant speech separation. Prior methods for unsupervised separation required the synthesis of mixtures of mixtures or assumed the existence of a teacher model, making them difficult to consider as potential methods explaining the emergence of separation abilities in an animal's auditory system. We assume the availability of two-channel mixtures at training time, and train a neural network to separate the sources given one of the channels as input such that the other channel may be predicted from the separated sources. As the relationship between the room impulse responses (RIRs) of each channel depends on the locations of the sources, which are unknown to the network, the network cannot rely on learning that relationship. Instead, our proposed loss function fits each of the separated sources to the mixture in the target channel via Wiener filtering, and compares the resulting mixture to the ground-truth one. We show that minimizing the scale-invariant signal-to-distortion ratio (SI-SDR) of the predicted right-channel mixture with respect to the ground truth implicitly guides the network towards separating the left-channel sources. On a semi-supervised reverberant speech separation task based on the WHAMR! dataset, using training data where just 5% (resp., 10%) of the mixtures are labeled with associated isolated sources, we achieve 70% (resp., 78%) of the SI-SDR improvement obtained when training with supervision on the full training set, while a model trained only on the labeled data obtains 43% (resp., 45%).