Towards an objective characterization of an individual's facial movements using Self-Supervised Person-Specific-Models

TL;DR

This paper introduces self-supervised person-specific models to independently characterize individual facial movements, enabling more precise and scalable analysis of facial dynamics across different people.

Contribution

The paper proposes a novel self-supervised training approach for person-specific facial movement models that are independent of facial identity, improving movement characterization.

Findings

Models learn meaningful facial movement embeddings

Approach is scalable and generalizable to new individuals

Temporal curriculum learning enhances model performance

Abstract

Disentangling facial movements from other facial characteristics, particularly from facial identity, remains a challenging task, as facial movements display great variation between individuals. In this paper, we aim to characterize individual-specific facial movements. We present a novel training approach to learn facial movements independently of other facial characteristics, focusing on each individual separately. We propose self-supervised Person-Specific Models (PSMs), in which one model per individual can learn to extract an embedding of the facial movements independently of the person's identity and other structural facial characteristics from unlabeled facial video. These models are trained using encoder-decoder-like architectures. We provide quantitative and qualitative evidence that a PSM learns a meaningful facial embedding that discovers fine-grained movements otherwise not characterized by a General Model (GM), which is trained across individuals and characterizes general patterns of facial movements. We present quantitative and qualitative evidence that this approach is easily scalable and generalizable for new individuals: facial movements knowledge learned on a person can quickly and effectively be transferred to a new person. Lastly, we propose a novel PSM using curriculum temporal learning to leverage the temporal contiguity between video frames. Our code, analysis details, and all pretrained models are available in Github and Supplementary Materials.