CausalAffect: Causal Discovery for Facial Affective Understanding

TL;DR

CausalAffect is a novel framework that discovers causal relationships among facial Action Units and expressions, improving affect recognition by integrating causal inference with facial behavior analysis.

Contribution

It introduces the first causal graph discovery method for facial affect analysis that does not require annotated causal data or priors, aligning with psychological theories.

Findings

Achieves state-of-the-art AU detection and expression recognition performance.

Recovers causal structures consistent with psychological theories.

Reveals novel inhibitory and uncharacterized dependencies.

Abstract

Understanding human affect from facial behavior requires not only accurate recognition but also structured reasoning over the latent dependencies that drive muscle activations and their expressive outcomes. Although Action Units (AUs) have long served as the foundation of affective computing, existing approaches rarely address how to infer psychologically plausible causal relations between AUs and expressions directly from data. We propose CausalAffect, the first framework for causal graph discovery in facial affect analysis. CausalAffect models AU-AU and AU-Expression dependencies through a two-level polarity and direction aware causal hierarchy that integrates population-level regularities with sample-adaptive structures. A feature-level counterfactual intervention mechanism further enforces true causal effects while suppressing spurious correlations. Crucially, our approach requires neither jointly annotated datasets nor handcrafted causal priors, yet it recovers causal structures consistent with established psychological theories while revealing novel inhibitory and previously uncharacterized dependencies. Extensive experiments across six benchmarks demonstrate that CausalAffect advances the state of the art in both AU detection and expression recognition, establishing a principled connection between causal discovery and interpretable facial behavior. All trained models and source code will be released upon acceptance.