Self-attention aggregation network for video face representation and recognition

TL;DR

This paper introduces a novel self-attention aggregation network (SAAN) for video face recognition that effectively handles multiple identities and outperforms traditional pooling methods, validated on public datasets.

Contribution

The paper presents the first aggregation approach considering multiple identities in videos using self-attention, enhancing face representation accuracy.

Findings

SAAN outperforms average pooling on IJB-C dataset.

SAAN effectively handles videos with multiple identities.

A new multi-identity video dataset was introduced.

Abstract

Models based on self-attention mechanisms have been successful in analyzing temporal data and have been widely used in the natural language domain. We propose a new model architecture for video face representation and recognition based on a self-attention mechanism. Our approach could be used for video with single and multiple identities. To the best of our knowledge, no one has explored the aggregation approaches that consider the video with multiple identities. The proposed approach utilizes existing models to get the face representation for each video frame, e.g., ArcFace and MobileFaceNet, and the aggregation module produces the aggregated face representation vector for video by taking into consideration the order of frames and their quality scores. We demonstrate empirical results on a public dataset for video face recognition called IJB-C to indicate that the self-attention aggregation network (SAAN) outperforms naive average pooling. Moreover, we introduce a new multi-identity video dataset based on the publicly available UMDFaces dataset and collected GIFs from Giphy. We show that SAAN is capable of producing a compact face representation for both single and multiple identities in a video. The dataset and source code will be publicly available.