Quality-Aware End-to-End Audio-Visual Neural Speaker Diarization

TL;DR

This paper introduces a comprehensive audio-visual speaker diarization system that effectively handles overlapping speech and signal degradations using a multi-modal, quality-aware, end-to-end neural framework with cross attention mechanisms.

Contribution

It presents a novel end-to-end audio-visual diarization model that incorporates quality-aware fusion and cross attention to improve robustness in challenging conditions.

Findings

Achieves high accuracy even with degraded video quality.

Effectively handles overlapping speech scenarios.

Demonstrates robustness across diverse acoustic environments.

Abstract

In this paper, we propose a quality-aware end-to-end audio-visual neural speaker diarization framework, which comprises three key techniques. First, our audio-visual model takes both audio and visual features as inputs, utilizing a series of binary classification output layers to simultaneously identify the activities of all speakers. This end-to-end framework is meticulously designed to effectively handle situations of overlapping speech, providing accurate discrimination between speech and non-speech segments through the utilization of multi-modal information. Next, we employ a quality-aware audio-visual fusion structure to address signal quality issues for both audio degradations, such as noise, reverberation and other distortions, and video degradations, such as occlusions, off-screen speakers, or unreliable detection. Finally, a cross attention mechanism applied to multi-speaker embedding empowers the network to handle scenarios with varying numbers of speakers. Our experimental results, obtained from various data sets, demonstrate the robustness of our proposed techniques in diverse acoustic environments. Even in scenarios with severely degraded video quality, our system attains performance levels comparable to the best available audio-visual systems.