Interpretable facial dynamics as behavioral and perceptual traces of deepfakes

TL;DR

This study introduces an interpretable, bio-behavioral approach to deepfake detection based on facial dynamics, revealing that emotional expressions significantly enhance detection accuracy and that model and human judgments often align.

Contribution

The paper proposes a bio-behavioral feature-based method for deepfake detection, linking computational strategies to human perception and highlighting the importance of emotional expressions.

Findings

Higher-order temporal irregularities are more pronounced in deepfakes.

Detection accuracy improves with videos showing emotional expressions.

Model and human judgments align more for emotive videos, diverge for non-emotive ones.

Abstract

Deepfake detection research has largely converged on deep learning approaches that, despite strong benchmark performance, offer limited insight into what distinguishes real from manipulated facial behavior. This study presents an interpretable alternative grounded in bio-behavioral features of facial dynamics and evaluates how computational detection strategies relate to human perceptual judgments. We identify core low-dimensional patterns of facial movement, from which temporal features characterizing spatiotemporal structure were derived. Traditional machine learning classifiers trained on these features achieved modest but significant above-chance deepfake classification, driven by higher-order temporal irregularities that were more pronounced in manipulated than real facial dynamics. Notably, detection was substantially more accurate for videos containing emotive expressions than those without. An emotional valence classification analysis further indicated that emotive signals are systematically degraded in deepfakes, explaining the differential impact of emotive dynamics on detection. Furthermore, we provide an additional and often overlooked dimension of explainability by assessing the relationship between model decisions and human perceptual detection. Model and human judgments converged for emotive but diverged for non-emotive videos, and even where outputs aligned, underlying detection strategies differed. These findings demonstrate that face-swapped deepfakes carry a measurable behavioral fingerprint, most salient during emotional expression. Additionally, model-human comparisons suggest that interpretable computational features and human perception may offer complementary rather than redundant routes to detection.