AU-vMAE: Knowledge-Guide Action Units Detection via Video Masked Autoencoder

TL;DR

AU-vMAE introduces a video-masked autoencoder pre-training scheme for facial action unit detection, leveraging multi-label video data and temporal consistency to improve performance over existing methods.

Contribution

The paper proposes a novel video-level pre-training approach using masked autoencoders and prior AU pair matrices, addressing data scarcity and diversity in facial action unit detection.

Findings

Significant performance improvements on BP4D and DISFA datasets.

Effective utilization of multi-label and temporal information.

Outperforms state-of-the-art methods in AU detection.

Abstract

Current Facial Action Unit (FAU) detection methods generally encounter difficulties due to the scarcity of labeled video training data and the limited number of training face IDs, which renders the trained feature extractor insufficient coverage for modeling the large diversity of inter-person facial structures and movements. To explicitly address the above challenges, we propose a novel video-level pre-training scheme by fully exploring the multi-label property of FAUs in the video as well as the temporal label consistency. At the heart of our design is a pre-trained video feature extractor based on the video-masked autoencoder together with a fine-tuning network that jointly completes the multi-level video FAUs analysis tasks, \emph{i.e.} integrating both video-level and frame-level FAU detections, thus dramatically expanding the supervision set from sparse FAUs annotations to ALL video frames including masked ones. Moreover, we utilize inter-frame and intra-frame AU pair state matrices as prior knowledge to guide network training instead of traditional Graph Neural Networks, for better temporal supervision. Our approach demonstrates substantial enhancement in performance compared to the existing state-of-the-art methods used in BP4D and DISFA FAUs datasets.