BIAS: A Body-based Interpretable Active Speaker Approach

TL;DR

BIAS introduces a novel, interpretable model that combines audio, face, and body cues to improve active speaker detection in challenging real-world scenarios, outperforming existing methods.

Contribution

The paper presents BIAS, the first model to integrate body-based features with audio and face data for ASD, enhancing performance and interpretability in wild conditions.

Findings

BIAS achieves state-of-the-art results in challenging datasets like WASD.

The model provides interpretability through attention heatmaps and feature importance.

BIAS performs competitively on standard datasets, emphasizing the importance of body cues.

Abstract

State-of-the-art Active Speaker Detection (ASD) approaches heavily rely on audio and facial features to perform, which is not a sustainable approach in wild scenarios. Although these methods achieve good results in the standard AVA-ActiveSpeaker set, a recent wilder ASD dataset (WASD) showed the limitations of such models and raised the need for new approaches. As such, we propose BIAS, a model that, for the first time, combines audio, face, and body information, to accurately predict active speakers in varying/challenging conditions. Additionally, we design BIAS to provide interpretability by proposing a novel use for Squeeze-and-Excitation blocks, namely in attention heatmaps creation and feature importance assessment. For a full interpretability setup, we annotate an ASD-related actions dataset (ASD-Text) to finetune a ViT-GPT2 for text scene description to complement BIAS interpretability. The results show that BIAS is state-of-the-art in challenging conditions where body-based features are of utmost importance (Columbia, open-settings, and WASD), and yields competitive results in AVA-ActiveSpeaker, where face is more influential than body for ASD. BIAS interpretability also shows the features/aspects more relevant towards ASD prediction in varying settings, making it a strong baseline for further developments in interpretable ASD models, and is available at https://github.com/Tiago-Roxo/BIAS.