Spectral or spatial? Leveraging both for speaker extraction in challenging data conditions

TL;DR

This paper introduces a multi-channel speaker extraction method that combines spectral and spatial cues to improve robustness in challenging, noisy, and reference-inaccurate conditions, outperforming existing approaches.

Contribution

It proposes a novel integrated approach that dynamically balances spectral and spatial information, ensuring stable speaker extraction despite reference inaccuracies.

Findings

Effective in noisy, reference-inaccurate scenarios

Outperforms existing methods in robustness

Maintains reliable performance with degraded cues

Abstract

This paper presents a robust multi-channel speaker extraction algorithm designed to handle inaccuracies in reference information. While existing approaches often rely solely on either spatial or spectral cues to identify the target speaker, our method integrates both sources of information to enhance robustness. A key aspect of our approach is its emphasis on stability, ensuring reliable performance even when one of the features is degraded or misleading. Given a noisy mixture and two potentially unreliable cues, a dedicated network is trained to dynamically balance their contributions-or disregard the less informative one when necessary. We evaluate the system under challenging conditions by simulating inference-time errors using a simple direction of arrival (DOA) estimator and a noisy spectral enrollment process. Experimental results demonstrate that the proposed model successfully extracts the desired speaker even in the presence of substantial reference inaccuracies.