Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings for Target Speaker Separation

TL;DR

This paper investigates overlooked aspects of enrollment speaker embeddings in target speaker separation, highlighting the effectiveness of filterbank embeddings over self-supervised ones and questioning the suitability of speaker identification embeddings.

Contribution

It introduces a comprehensive analysis of various enrollment embeddings, revealing the superiority of filterbank embeddings for cross-dataset generalization in TSS.

Findings

Filterbank embeddings outperform self-supervised embeddings in cross-dataset tests.

Speaker identification embeddings may lose relevant information due to sub-optimal metrics or training objectives.

Filterbank embeddings consistently show competitive separation and generalization performance.

Abstract

Single channel target speaker separation (TSS) aims at extracting a speaker's voice from a mixture of multiple talkers given an enrollment utterance of that speaker. A typical deep learning TSS framework consists of an upstream model that obtains enrollment speaker embeddings and a downstream model that performs the separation conditioned on the embeddings. In this paper, we look into several important but overlooked aspects of the enrollment embeddings, including the suitability of the widely used speaker identification embeddings, the introduction of the log-mel filterbank and self-supervised embeddings, and the embeddings' cross-dataset generalization capability. Our results show that the speaker identification embeddings could lose relevant information due to a sub-optimal metric, training objective, or common pre-processing. In contrast, both the filterbank and the self-supervised embeddings preserve the integrity of the speaker information, but the former consistently outperforms the latter in a cross-dataset evaluation. The competitive separation and generalization performance of the previously overlooked filterbank embedding is consistent across our study, which calls for future research on better upstream features.