FabuLight-ASD: Unveiling Speech Activity via Body Language

TL;DR

FabuLight-ASD enhances active speaker detection by integrating facial, audio, and body pose data, achieving higher accuracy with minimal computational overhead, especially in challenging scenarios like occlusion and background noise.

Contribution

This work introduces FabuLight-ASD, a novel ASD model that combines body pose information with existing methods to improve detection accuracy and robustness.

Findings

Achieved 94.3% mAP, outperforming Light-ASD's 93.7%.

Significant improvements in occlusion and noise scenarios.

Modest increase in computational complexity (27.3% more parameters).

Abstract

Active speaker detection (ASD) in multimodal environments is crucial for various applications, from video conferencing to human-robot interaction. This paper introduces FabuLight-ASD, an advanced ASD model that integrates facial, audio, and body pose information to enhance detection accuracy and robustness. Our model builds upon the existing Light-ASD framework by incorporating human pose data, represented through skeleton graphs, which minimises computational overhead. Using the Wilder Active Speaker Detection (WASD) dataset, renowned for reliable face and body bounding box annotations, we demonstrate FabuLight-ASD's effectiveness in real-world scenarios. Achieving an overall mean average precision (mAP) of 94.3%, FabuLight-ASD outperforms Light-ASD, which has an overall mAP of 93.7% across various challenging scenarios. The incorporation of body pose information shows a particularly advantageous impact, with notable improvements in mAP observed in scenarios with speech impairment, face occlusion, and human voice background noise. Furthermore, efficiency analysis indicates only a modest increase in parameter count (27.3%) and multiply-accumulate operations (up to 2.4%), underscoring the model's efficiency and feasibility. These findings validate the efficacy of FabuLight-ASD in enhancing ASD performance through the integration of body pose data. FabuLight-ASD's code and model weights are available at https://github.com/knowledgetechnologyuhh/FabuLight-ASD.