Frame-wise streaming end-to-end speaker diarization with non-autoregressive self-attention-based attractors

TL;DR

This paper introduces a novel frame-wise online speaker diarization method using non-autoregressive self-attention to detect and update multiple speakers in real time with low latency and high accuracy.

Contribution

It presents a new frame-wise streaming end-to-end neural diarization approach utilizing a causal encoder and look-ahead self-attention decoder for real-time speaker tracking.

Findings

Achieves state-of-the-art diarization accuracy.

Maintains low inference latency and computational cost.

Effectively detects new speakers in real time.

Abstract

This work proposes a frame-wise online/streaming end-to-end neural diarization (FS-EEND) method in a frame-in-frame-out fashion. To frame-wisely detect a flexible number of speakers and extract/update their corresponding attractors, we propose to leverage a causal speaker embedding encoder and an online non-autoregressive self-attention-based attractor decoder. A look-ahead mechanism is adopted to allow leveraging some future frames for effectively detecting new speakers in real time and adaptively updating speaker attractors. The proposed method processes the audio stream frame by frame, and has a low inference latency caused by the look-ahead frames. Experiments show that, compared with the recently proposed block-wise online methods, our method FS-EEND achieves state-of-the-art diarization results, with a low inference latency and computational cost.