An Empirical Analysis on the Vulnerabilities of End-to-End Speech Segregation Models

TL;DR

This paper investigates the mechanisms of end-to-end speech segregation models, revealing their reliance on harmonic cues, their instability under certain conditions, and the impact of encoder design on their performance and robustness.

Contribution

It provides a detailed analysis of how ConvTasnet and DPT-Net perform harmonic analysis, identifies sources of errors, and suggests that replacing the encoder improves stability.

Findings

End-to-end models are highly unstable with imperceptible deformations.

Replacing the encoder with a spectrogram reduces performance but increases stability.

Harmonic cues are critical for speech segregation in these models.

Abstract

End-to-end learning models have demonstrated a remarkable capability in performing speech segregation. Despite their wide-scope of real-world applications, little is known about the mechanisms they employ to group and consequently segregate individual speakers. Knowing that harmonicity is a critical cue for these networks to group sources, in this work, we perform a thorough investigation on ConvTasnet and DPT-Net to analyze how they perform a harmonic analysis of the input mixture. We perform ablation studies where we apply low-pass, high-pass, and band-stop filters of varying pass-bands to empirically analyze the harmonics most critical for segregation. We also investigate how these networks decide which output channel to assign to an estimated source by introducing discontinuities in synthetic mixtures. We find that end-to-end networks are highly unstable, and perform poorly when confronted with deformations which are imperceptible to humans. Replacing the encoder in these networks with a spectrogram leads to lower overall performance, but much higher stability. This work helps us to understand what information these network rely on for speech segregation, and exposes two sources of generalization-errors. It also pinpoints the encoder as the part of the network responsible for these errors, allowing for a redesign with expert knowledge or transfer learning.