Unsupervised active speaker detection in media content using cross-modal information

TL;DR

This paper introduces an unsupervised, cross-modal framework for detecting active speakers in media content by matching speech segments with facial images based on speaker identity, without requiring labeled data.

Contribution

It formulates active speaker detection as a speech-face assignment problem leveraging speaker identity distances, addressing off-screen speakers, and achieves competitive results on multiple datasets.

Findings

Competitive performance to supervised methods on benchmark datasets

Effective handling of off-screen speakers

Unsupervised approach reduces need for labeled data

Abstract

We present a cross-modal unsupervised framework for active speaker detection in media content such as TV shows and movies. Machine learning advances have enabled impressive performance in identifying individuals from speech and facial images. We leverage speaker identity information from speech and faces, and formulate active speaker detection as a speech-face assignment task such that the active speaker's face and the underlying speech identify the same person (character). We express the speech segments in terms of their associated speaker identity distances, from all other speech segments, to capture a relative identity structure for the video. Then we assign an active speaker's face to each speech segment from the concurrently appearing faces such that the obtained set of active speaker faces displays a similar relative identity structure. Furthermore, we propose a simple and effective approach to address speech segments where speakers are present off-screen. We evaluate the proposed system on three benchmark datasets -- Visual Person Clustering dataset, AVA-active speaker dataset, and Columbia dataset -- consisting of videos from entertainment and broadcast media, and show competitive performance to state-of-the-art fully supervised methods.