Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalities

TL;DR

This paper introduces GLDTracker, a novel audio-visual speaker tracking system that effectively handles incomplete modalities using a teacher-student distillation framework and global-local feature reconstruction, improving robustness in complex scenes.

Contribution

The paper proposes a global-local distillation network that fuses incomplete multi-modal data for robust speaker tracking, a novel approach in handling missing modalities.

Findings

Outperforms existing state-of-the-art trackers on AV16.3 dataset.

Demonstrates robustness with incomplete audio-visual data.

Achieves superior accuracy in complex dynamic scenes.

Abstract

In speaker tracking research, integrating and complementing multi-modal data is a crucial strategy for improving the accuracy and robustness of tracking systems. However, tracking with incomplete modalities remains a challenging issue due to noisy observations caused by occlusion, acoustic noise, and sensor failures. Especially when there is missing data in multiple modalities, the performance of existing multi-modal fusion methods tends to decrease. To this end, we propose a Global-Local Distillation-based Tracker (GLDTracker) for robust audio-visual speaker tracking. GLDTracker is driven by a teacher-student distillation model, enabling the flexible fusion of incomplete information from each modality. The teacher network processes global signals captured by camera and microphone arrays, and the student network handles local information subject to visual occlusion and missing audio channels. By transferring knowledge from teacher to student, the student network can better adapt to complex dynamic scenes with incomplete observations. In the student network, a global feature reconstruction module based on the generative adversarial network is constructed to reconstruct global features from feature embedding with missing local information. Furthermore, a multi-modal multi-level fusion attention is introduced to integrate the incomplete feature and the reconstructed feature, leveraging the complementarity and consistency of audio-visual and global-local features. Experimental results on the AV16.3 dataset demonstrate that the proposed GLDTracker outperforms existing state-of-the-art audio-visual trackers and achieves leading performance on both standard and incomplete modalities datasets, highlighting its superiority and robustness in complex conditions. The code and models will be available.