Self-supervised Video-centralised Transformer for Video Face Clustering

TL;DR

This paper introduces a self-supervised, video-centralised transformer for face clustering in videos, effectively capturing temporal dynamics and outperforming previous methods on standard and egocentric datasets.

Contribution

The paper proposes a novel self-supervised transformer framework for video face clustering, including the first large-scale egocentric dataset, improving clustering accuracy over existing approaches.

Findings

Outperforms state-of-the-art on BBT dataset

Achieves superior results on EasyCom-Clustering dataset

Effectively captures temporal face dynamics

Abstract

This paper presents a novel method for face clustering in videos using a video-centralised transformer. Previous works often employed contrastive learning to learn frame-level representation and used average pooling to aggregate the features along the temporal dimension. This approach may not fully capture the complicated video dynamics. In addition, despite the recent progress in video-based contrastive learning, few have attempted to learn a self-supervised clustering-friendly face representation that benefits the video face clustering task. To overcome these limitations, our method employs a transformer to directly learn video-level representations that can better reflect the temporally-varying property of faces in videos, while we also propose a video-centralised self-supervised framework to train the transformer model. We also investigate face clustering in egocentric videos, a fast-emerging field that has not been studied yet in works related to face clustering. To this end, we present and release the first large-scale egocentric video face clustering dataset named EasyCom-Clustering. We evaluate our proposed method on both the widely used Big Bang Theory (BBT) dataset and the new EasyCom-Clustering dataset. Results show the performance of our video-centralised transformer has surpassed all previous state-of-the-art methods on both benchmarks, exhibiting a self-attentive understanding of face videos.