Detection and Localization of Facial Expression Manipulations

TL;DR

This paper introduces a novel framework combining facial expression recognition and image manipulation techniques to detect and localize facial expression manipulations with higher accuracy than existing methods.

Contribution

The proposed method effectively detects and localizes facial expression manipulations by integrating expression recognition features, outperforming state-of-the-art approaches on multiple datasets.

Findings

Over 3% higher accuracy on Face2Face dataset

Over 2% higher accuracy on NeuralTextures dataset

Performs comparably in identity change scenarios

Abstract

Concern regarding the wide-spread use of fraudulent images/videos in social media necessitates precise detection of such fraud. The importance of facial expressions in communication is widely known, and adversarial attacks often focus on manipulating the expression related features. Thus, it is important to develop methods that can detect manipulations in facial expressions, and localize the manipulated regions. To address this problem, we propose a framework that is able to detect manipulations in facial expression using a close combination of facial expression recognition and image manipulation methods. With the addition of feature maps extracted from the facial expression recognition framework, our manipulation detector is able to localize the manipulated region. We show that, on the Face2Face dataset, where there is abundant expression manipulation, our method achieves over 3% higher accuracy for both classification and localization of manipulations compared to state-of-the-art methods. In addition, results on the NeuralTextures dataset where the facial expressions corresponding to the mouth regions have been modified, show 2% higher accuracy in both classification and localization of manipulation. We demonstrate that the method performs at-par with the state-of-the-art methods in cases where the expression is not manipulated, but rather the identity is changed, thus ensuring generalizability of the approach.