EgoVIS@CVPR: PAIR-Net: Enhancing Egocentric Speaker Detection via Pretrained Audio-Visual Fusion and Alignment Loss

TL;DR

This paper presents PAIR-Net, a novel audio-visual fusion model with alignment loss that significantly improves active speaker detection in challenging egocentric videos, achieving state-of-the-art results.

Contribution

Introducing PAIR-Net, which combines pretrained audio and visual encoders with an alignment loss for robust egocentric speaker detection.

Findings

Achieves 76.6% mAP on Ego4D ASD benchmark.

Surpasses previous methods LoCoNet and STHG by 8.2% and 12.9% mAP.

Demonstrates effectiveness of pretrained audio priors and alignment in real-world conditions.

Abstract

Active speaker detection (ASD) in egocentric videos presents unique challenges due to unstable viewpoints, motion blur, and off-screen speech sources - conditions under which traditional visual-centric methods degrade significantly. We introduce PAIR-Net (Pretrained Audio-Visual Integration with Regularization Network), an effective model that integrates a partially frozen Whisper audio encoder with a fine-tuned AV-HuBERT visual backbone to robustly fuse cross-modal cues. To counteract modality imbalance, we introduce an inter-modal alignment loss that synchronizes audio and visual representations, enabling more consistent convergence across modalities. Without relying on multi-speaker context or ideal frontal views, PAIR-Net achieves state-of-the-art performance on the Ego4D ASD benchmark with 76.6% mAP, surpassing LoCoNet and STHG by 8.2% and 12.9% mAP, respectively. Our results highlight the value of pretrained audio priors and alignment-based fusion for robust ASD under real-world egocentric conditions.