Occlusion-Robust FAU Recognition by Mining Latent Space of Masked Autoencoders

TL;DR

This paper introduces an occlusion-robust FAU recognition method that leverages the latent space of masked autoencoders, enabling accurate facial action unit detection even under heavy occlusions, outperforming existing methods.

Contribution

The paper presents the first occlusion-robust FAU recognition approach using latent space mining of masked autoencoders, bypassing occlusion reconstruction for improved efficiency.

Findings

Achieves state-of-the-art performance under heavy occlusion conditions.

Outperforms baseline methods on BP4D and DISFA datasets.

Maintains comparable accuracy to normal-condition methods despite occlusions.

Abstract

Facial action units (FAUs) are critical for fine-grained facial expression analysis. Although FAU detection has been actively studied using ideally high quality images, it was not thoroughly studied under heavily occluded conditions. In this paper, we propose the first occlusion-robust FAU recognition method to maintain FAU detection performance under heavy occlusions. Our novel approach takes advantage of rich information from the latent space of masked autoencoder (MAE) and transforms it into FAU features. Bypassing the occlusion reconstruction step, our model efficiently extracts FAU features of occluded faces by mining the latent space of a pretrained masked autoencoder. Both node and edge-level knowledge distillation are also employed to guide our model to find a mapping between latent space vectors and FAU features. Facial occlusion conditions, including random small patches and large blocks, are thoroughly studied. Experimental results on BP4D and DISFA datasets show that our method can achieve state-of-the-art performances under the studied facial occlusion, significantly outperforming existing baseline methods. In particular, even under heavy occlusion, the proposed method can achieve comparable performance as state-of-the-art methods under normal conditions.