An Investigation on Speaker Augmentation for End-to-End Speaker Extraction

TL;DR

This paper introduces a speaker augmentation method using time-domain resampling and rescaling to improve speaker embeddings, reducing target confusion and enhancing end-to-end speaker extraction performance.

Contribution

It proposes a novel augmentation strategy that enhances generalizability and discrimination of speaker embeddings in E2E-SE systems, addressing target confusion.

Findings

Reduces target confusion in speaker extraction

Improves performance on WSJ0-2Mix and LibriMix datasets

Combines effectively with metric learning for further gains

Abstract

Target confusion, defined as occasional switching to non-target speakers, poses a key challenge for end-to-end speaker extraction (E2E-SE) systems. We argue that this problem is largely caused by the lack of generalizability and discrimination of the speaker embeddings, and introduce a simple yet effective speaker augmentation strategy to tackle the problem. Specifically, we propose a time-domain resampling and rescaling pipeline that alters speaker traits while preserving other speech properties. This generates a variety of pseudo-speakers to help establish a generalizable speaker embedding space, while the speaker-trait-specific augmentation creates hard samples that force the model to focus on genuine speaker characteristics. Experiments on WSJ0-2Mix and LibriMix show that our method mitigates the target confusion and improves extraction performance. Moreover, it can be combined with metric learning, another effective approach to address target confusion, leading to further gains.