Investigation of Speaker Representation for Target-Speaker Speech Processing

TL;DR

This study compares various speaker embeddings for target-speaker speech processing tasks, revealing that ideal embeddings differ from verification performance and that one-hot vectors often outperform enrollment-based embeddings.

Contribution

It provides a comprehensive cross-task evaluation of speaker embeddings, highlighting the effectiveness of one-hot vectors and optimizing embeddings for improved target-speaker processing.

Findings

One-hot vectors outperform enrollment-based embeddings.

Verification performance is not strongly correlated with TS task performance.

Optimized embeddings vary depending on input mixture characteristics.

Abstract

Target-speaker speech processing (TS) tasks, such as target-speaker automatic speech recognition (TS-ASR), target speech extraction (TSE), and personal voice activity detection (p-VAD), are important for extracting information about a desired speaker's speech even when it is corrupted by interfering speakers. While most studies have focused on training schemes or system architectures for each specific task, the auxiliary network for embedding target-speaker cues has not been investigated comprehensively in a unified cross-task evaluation. Therefore, this paper aims to address a fundamental question: what is the preferred speaker embedding for TS tasks? To this end, for the TS-ASR, TSE, and p-VAD tasks, we compare pre-trained speaker encoders (i.e., self-supervised or speaker recognition models) that compute speaker embeddings from pre-recorded enrollment speech of the target speaker with ideal speaker embeddings derived directly from the target speaker's identity in the form of a one-hot vector. To further understand the properties of ideal speaker embedding, we optimize it using a gradient-based approach to improve performance on the TS task. Our analysis reveals that speaker verification performance is somewhat unrelated to TS task performances, the one-hot vector outperforms enrollment-based ones, and the optimal embedding depends on the input mixture.