LS-EEND: Long-Form Streaming End-to-End Neural Diarization with Online Attractor Extraction

TL;DR

This paper introduces LS-EEND, a novel online neural diarization method capable of handling long recordings and multiple speakers with high accuracy and efficiency, advancing real-time speaker diarization technology.

Contribution

The paper presents a new streaming EEND model with online attractor extraction, a retention mechanism for long audio, and a progressive training strategy, enabling high-performance diarization for lengthy recordings.

Findings

Achieves state-of-the-art online diarization error rates on multiple datasets.

Handles up to 8 speakers and hour-long recordings effectively.

Operates several times faster than existing online diarization models.

Abstract

This work proposes a frame-wise online/streaming end-to-end neural diarization (EEND) method, which detects speaker activities in a frame-in-frame-out fashion. The proposed model mainly consists of a causal embedding encoder and an online attractor decoder. Speakers are modeled in the self-attention-based decoder along both the time and speaker dimensions, and frame-wise speaker attractors are automatically generated and updated for new speakers and existing speakers, respectively. Retention mechanism is employed and especially adapted for long-form diarization with a linear temporal complexity. A multi-step progressive training strategy is proposed for gradually learning from easy tasks to hard tasks in terms of the number of speakers and audio length. Finally, the proposed model (referred to as long-form streaming EEND, LS-EEND) is able to perform streaming diarization for a high (up to 8) and flexible number speakers and very long (say one hour) audio recordings. Experiments on various simulated and real-world datasets show that: 1) when not using oracle speech activity information, the proposed model achieves new state-of-the-art online diarization error rate on all datasets, including CALLHOME (12.11%), DIHARD II (27.58%), DIHARD III (19.61%), and AMI (20.76%); 2) Due to the frame-in-frame-out processing fashion and the linear temporal complexity, the proposed model achieves several times lower real-time-factor than comparison online diarization models.