Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Network

TL;DR

This paper introduces WEEND, a multi-task neural network that performs simultaneous speech recognition and speaker diarization at the word level, streamlining the process and improving accuracy in multi-speaker scenarios.

Contribution

The paper presents a novel end-to-end neural architecture with auxiliary networks for joint speech recognition and speaker diarization, enabling real-time, word-level labeling.

Findings

Outperforms baseline on 2-speaker scenarios

Generalizes to 5-minute audio segments

Potential for high-quality diarization with sufficient training data

Abstract

While standard speaker diarization attempts to answer the question "who spoken when", most of relevant applications in reality are more interested in determining "who spoken what". Whether it is the conventional modularized approach or the more recent end-to-end neural diarization (EEND), an additional automatic speech recognition (ASR) model and an orchestration algorithm are required to associate the speaker labels with recognized words. In this paper, we propose Word-level End-to-End Neural Diarization (WEEND) with auxiliary network, a multi-task learning algorithm that performs end-to-end ASR and speaker diarization in the same neural architecture. That is, while speech is being recognized, speaker labels are predicted simultaneously for each recognized word. Experimental results demonstrate that WEEND outperforms the turn-based diarization baseline system on all 2-speaker short-form scenarios and has the capability to generalize to audio lengths of 5 minutes. Although 3+speaker conversations are harder, we find that with enough in-domain training data, WEEND has the potential to deliver high quality diarized text.