EEND-M2F: Masked-attention mask transformers for speaker diarization

TL;DR

This paper introduces EEND-M2F, a novel end-to-end speaker diarization model inspired by image segmentation, achieving state-of-the-art results without clustering or additional models.

Contribution

The paper proposes EEND-M2F, a lightweight, fully end-to-end diarization model based on Mask2Former architecture, improving performance on multiple datasets.

Findings

Achieves 16.07% DER on DIHARD-III, surpassing previous systems.

Does not require clustering or additional speaker verification models.

Demonstrates efficiency and effectiveness across several public datasets.

Abstract

In this paper, we make the explicit connection between image segmentation methods and end-to-end diarization methods. From these insights, we propose a novel, fully end-to-end diarization model, EEND-M2F, based on the Mask2Former architecture. Speaker representations are computed in parallel using a stack of transformer decoders, in which irrelevant frames are explicitly masked from the cross attention using predictions from previous layers. EEND-M2F is lightweight, efficient, and truly end-to-end, as it does not require any additional diarization, speaker verification, or segmentation models to run, nor does it require running any clustering algorithms. Our model achieves state-of-the-art performance on several public datasets, such as AMI, AliMeeting and RAMC. Most notably our DER of 16.07% on DIHARD-III is the first major improvement upon the challenge winning system.