Universal Speaker Embedding Free Target Speaker Extraction and Personal Voice Activity Detection

TL;DR

This paper introduces USEF-TP, a novel model that jointly performs target speaker extraction and personal voice activity detection without relying on speaker embeddings, improving robustness across diverse scenarios.

Contribution

The paper proposes a universal, embedding-free approach for TSE and PVAD using cross-attention and multi-task learning, addressing inconsistencies in traditional methods.

Findings

Achieves superior TSE and PVAD performance on LibriMix and SparseLibriMix datasets.

Demonstrates competitive results on real-world CALLHOME recordings.

Outperforms existing methods in handling overlapping speakers.

Abstract

Determining 'who spoke what and when' remains challenging in real-world applications. In typical scenarios, Speaker Diarization (SD) is employed to address the problem of 'who spoke when,' while Target Speaker Extraction (TSE) or Target Speaker Automatic Speech Recognition (TSASR) techniques are utilized to resolve the issue of 'who spoke what.' Although some works have achieved promising results by combining SD and TSE systems, inconsistencies remain between SD and TSE regarding both output inconsistency and scenario mismatch. To address these limitations, we propose a Universal Speaker Embedding Free Target Speaker Extraction and Personal Voice Activity Detection (USEF-TP) model that jointly performs TSE and Personal Voice Activity Detection (PVAD). USEF-TP leverages frame-level features obtained through a cross-attention mechanism as speaker-related features instead of using speaker embeddings as in traditional approaches. Additionally, a multi-task learning algorithm with a scenario-aware differentiated loss function is applied to ensure robust performance across various levels of speaker overlap. The experimental results show that our proposed USEF-TP model achieves superior performance in TSE and PVAD tasks on the LibriMix and SparseLibriMix datasets. The results on the CALLHOME dataset demonstrate the competitive performance of our model on real recordings.