Dual-Stream Decoupled Learning for Temporal Consistency and Speaker Interaction in AVSD

TL;DR

This paper introduces D$^2$Stream, a dual-stream framework that decouples temporal and social cues for improved audio-visual speaker detection, achieving state-of-the-art results.

Contribution

The paper proposes a novel decoupled dual-stream architecture that isolates temporal and social features, addressing conflicting biases in AVSD modeling.

Findings

Achieves 95.6% mAP on AVA-ActiveSpeaker

Demonstrates effective task decoupling through gradient analysis

Outperforms previous methods on Columbia ASD

Abstract

Audio-Visual Speaker Detection (AVSD) hinges on modeling both individual temporal continuity and inter-personal social context. Existing coupled architectures struggle to reconcile these tasks in shared representation spaces due to conflicting inductive biases: temporal modeling favors low-frequency smoothness, while inter-personal interaction requires high-frequency discriminability. We propose D$^2$Stream, a decoupled dual-stream framework that explicitly isolates these functionalities into parallel, task-specific branches. Specifically, the Intra-speaker Temporal Continuity (ITC) stream captures longitudinal stability, whereas the Inter-personal Social Relation (ISR) stream models transversal social cues. Quantitative gradient analysis reveals an evolutionary divergence in update directions, stabilizing at 86.1{\deg}, which confirms the inherent task conflict and the effectiveness of our structural decoupling. D$^2$Stream breaks the long-standing performance plateau, achieving a state-of-the-art 95.6% mAP on AVA-ActiveSpeaker and superior generalization on Columbia ASD, all within a lightweight and efficient design.