Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample

TL;DR

This paper introduces AC2D, a novel facial action unit detection framework that adaptively constrains self-attention and causally deconfounds sample bias, improving detection accuracy on challenging benchmarks.

Contribution

The paper proposes a new AU detection method that adaptively constrains self-attention and employs causal intervention to reduce bias and irrelevant information.

Findings

Achieves competitive performance on BP4D, DISFA, GFT, BP4D+, and Aff-Wild2 datasets.

Effectively suppresses sample bias and irrelevant AU interference.

Outperforms existing state-of-the-art AU detection methods.

Abstract

Facial action unit (AU) detection remains a challenging task, due to the subtlety, dynamics, and diversity of AUs. Recently, the prevailing techniques of self-attention and causal inference have been introduced to AU detection. However, most existing methods directly learn self-attention guided by AU detection, or employ common patterns for all AUs during causal intervention. The former often captures irrelevant information in a global range, and the latter ignores the specific causal characteristic of each AU. In this paper, we propose a novel AU detection framework called AC2D by adaptively constraining self-attention weight distribution and causally deconfounding the sample confounder. Specifically, we explore the mechanism of self-attention weight distribution, in which the self-attention weight distribution of each AU is regarded as spatial distribution and is adaptively learned under the constraint of location-predefined attention and the guidance of AU detection. Moreover, we propose a causal intervention module for each AU, in which the bias caused by training samples and the interference from irrelevant AUs are both suppressed. Extensive experiments show that our method achieves competitive performance compared to state-of-the-art AU detection approaches on challenging benchmarks, including BP4D, DISFA, GFT, and BP4D+ in constrained scenarios and Aff-Wild2 in unconstrained scenarios. The code is available at https://github.com/ZhiwenShao/AC2D.