ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection

TL;DR

ASDnB introduces a novel active speaker detection model that combines face and body cues, leveraging efficient 3D convolution splitting and adaptive feature weighting to improve robustness in diverse and challenging conditions.

Contribution

The paper presents a new model that integrates face and body information for active speaker detection, with a novel architecture and training strategy that enhances performance in difficult scenarios.

Findings

Achieves state-of-the-art results on AVA-ActiveSpeaker dataset.

Performs well in challenging datasets like WASD and cross-domain settings.

Offers a computationally efficient approach with split 3D convolutions.

Abstract

State-of-the-art Active Speaker Detection (ASD) approaches mainly use audio and facial features as input. However, the main hypothesis in this paper is that body dynamics is also highly correlated to "speaking" (and "listening") actions and should be particularly useful in wild conditions (e.g., surveillance settings), where face cannot be reliably accessed. We propose ASDnB, a model that singularly integrates face with body information by merging the inputs at different steps of feature extraction. Our approach splits 3D convolution into 2D and 1D to reduce computation cost without loss of performance, and is trained with adaptive weight feature importance for improved complement of face with body data. Our experiments show that ASDnB achieves state-of-the-art results in the benchmark dataset (AVA-ActiveSpeaker), in the challenging data of WASD, and in cross-domain settings using Columbia. This way, ASDnB can perform in multiple settings, which is positively regarded as a strong baseline for robust ASD models (code available at https://github.com/Tiago-Roxo/ASDnB).