Facing Asymmetry -- Uncovering the Causal Link between Facial Symmetry and Expression Classifiers using Synthetic Interventions

TL;DR

This paper investigates how facial symmetry causally influences the decisions of expression classifiers, revealing that reduced symmetry consistently lowers classifier outputs, with implications for understanding model behavior on out-of-distribution data.

Contribution

It introduces a causal framework using synthetic interventions to analyze the impact of facial symmetry on black-box expression classifiers, a novel approach in this context.

Findings

All classifiers show decreased output with reduced facial symmetry

The causal analysis aligns with real-world observations on facial palsy patients

Provides a case study for causal factors affecting black-box model behavior

Abstract

Understanding expressions is vital for deciphering human behavior, and nowadays, end-to-end trained black box models achieve high performance. Due to the black-box nature of these models, it is unclear how they behave when applied out-of-distribution. Specifically, these models show decreased performance for unilateral facial palsy patients. We hypothesize that one crucial factor guiding the internal decision rules is facial symmetry. In this work, we use insights from causal reasoning to investigate the hypothesis. After deriving a structural causal model, we develop a synthetic interventional framework. This approach allows us to analyze how facial symmetry impacts a network's output behavior while keeping other factors fixed. All 17 investigated expression classifiers significantly lower their output activations for reduced symmetry. This result is congruent with observed behavior on real-world data from healthy subjects and facial palsy patients. As such, our investigation serves as a case study for identifying causal factors that influence the behavior of black-box models.