LoCoNet: Long-Short Context Network for Active Speaker Detection

TL;DR

LoCoNet introduces a novel neural network architecture that effectively models both long-term intra-speaker and short-term inter-speaker contexts for active speaker detection, achieving state-of-the-art results across multiple datasets.

Contribution

The paper proposes LoCoNet, a simple yet effective model combining self-attention and convolutional blocks to jointly model long- and short-term speaker contexts for improved ASD performance.

Findings

Achieves state-of-the-art mAP scores on multiple datasets.

Outperforms previous methods in challenging multi-speaker scenarios.

Demonstrates significant improvements in detecting small or multiple active speakers.

Abstract

Active Speaker Detection (ASD) aims to identify who is speaking in each frame of a video. ASD reasons from audio and visual information from two contexts: long-term intra-speaker context and short-term inter-speaker context. Long-term intra-speaker context models the temporal dependencies of the same speaker, while short-term inter-speaker context models the interactions of speakers in the same scene. These two contexts are complementary to each other and can help infer the active speaker. Motivated by these observations, we propose LoCoNet, a simple yet effective Long-Short Context Network that models the long-term intra-speaker context and short-term inter-speaker context. We use self-attention to model long-term intra-speaker context due to its effectiveness in modeling long-range dependencies, and convolutional blocks that capture local patterns to model short-term inter-speaker context. Extensive experiments show that LoCoNet achieves state-of-the-art performance on multiple datasets, achieving an mAP of 95.2%(+1.1%) on AVA-ActiveSpeaker, 68.1%(+22%) on Columbia dataset, 97.2%(+2.8%) on Talkies dataset and 59.7%(+8.0%) on Ego4D dataset. Moreover, in challenging cases where multiple speakers are present, or face of active speaker is much smaller than other faces in the same scene, LoCoNet outperforms previous state-of-the-art methods by 3.4% on the AVA-ActiveSpeaker dataset. The code will be released at https://github.com/SJTUwxz/LoCoNet_ASD.