On the Use of Self-Supervised Representation Learning for Speaker Diarization and Separation

TL;DR

This paper evaluates the effectiveness of recent self-supervised speech models like wav2vec2.0 and WavLM on speaker diarization and separation tasks, addressing gaps in existing benchmarks and highlighting their potential in low-resource scenarios.

Contribution

It provides a comprehensive assessment of self-supervised speech representations on diarization and separation, revealing current limitations and proposing directions for more diverse evaluations.

Findings

Self-supervised models improve speaker diarization and separation performance.

Current benchmarks lack diversity in evaluation datasets.

There are significant gaps in evaluating these models on real-world tasks.

Abstract

Self-supervised speech models such as wav2vec2.0 and WavLM have been shown to significantly improve the performance of many downstream speech tasks, especially in low-resource settings, over the past few years. Despite this, evaluations on tasks such as Speaker Diarization and Speech Separation remain limited. This paper investigates the quality of recent self-supervised speech representations on these two speaker identity-related tasks, highlighting gaps in the current literature that stem from limitations in the existing benchmarks, particularly the lack of diversity in evaluation datasets and variety in downstream systems associated to both diarization and separation.