EvoTSE: Evolving Enrollment for Target Speaker Extraction

TL;DR

EvoTSE introduces an evolving enrollment mechanism for target speaker extraction that updates over time to reduce speaker confusion and improve performance, especially in out-of-domain scenarios.

Contribution

It proposes a novel evolving TSE framework that updates enrollment dynamically, overcoming static limitations and reducing the need for high-quality pre-recorded enrollment.

Findings

EvoTSE achieves consistent improvements across multiple benchmarks.

The method performs particularly well on out-of-domain scenarios.

Dynamic enrollment reduces speaker confusion effectively.

Abstract

Target Speaker Extraction (TSE) aims to isolate a specific speaker's voice from a mixture, guided by a pre-recorded enrollment. While TSE bypasses the global permutation ambiguity of blind source separation, it remains vulnerable to speaker confusion, where models mistakenly extract the interfering speaker. Furthermore, conventional TSE relies on static inference pipeline, where performance is limited by the quality of the fixed enrollment. To overcome these limitations, we propose EvoTSE, an evolving TSE framework in which the enrollment is continuously updated through reliability-filtered retrieval over high-confidence historical estimates. This mechanism reduces speaker confusion and relaxes the quality requirements for pre-recorded enrollment without relying on additional annotated data. Experiments across multiple benchmarks demonstrate that EvoTSE achieves consistent improvements, especially when evaluated on out-of-domain (OOD) scenarios. Our code and checkpoints are available.