Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Method

TL;DR

This paper redefines face forgery by its semantic alterations, introduces a hierarchical dataset and testing protocols, and proposes a semantics-oriented detection method that improves generalizability and detection accuracy.

Contribution

It provides a new semantic-based definition of face forgery, a hierarchical dataset with testing protocols, and a novel detection method emphasizing semantic label relations.

Findings

The dataset exposes weaknesses of current detectors.

The semantics-oriented method outperforms traditional detectors.

Improved generalizability of face forgery detection.

Abstract

In recent years, deep learning has greatly streamlined the process of manipulating photographic face images. Aware of the potential dangers, researchers have developed various tools to spot these counterfeits. Yet, none asks the fundamental question: What digital manipulations make a real photographic face image fake, while others do not? In this paper, we put face forgery in a semantic context and define that computational methods that alter semantic face attributes to exceed human discrimination thresholds are sources of face forgery. Following our definition, we construct a large face forgery image dataset, where each image is associated with a set of labels organized in a hierarchical graph. Our dataset enables two new testing protocols to probe the generalizability of face forgery detectors. Moreover, we propose a semantics-oriented face forgery detection method that captures label relations and prioritizes the primary task (i.e., real or fake face detection). We show that the proposed dataset successfully exposes the weaknesses of current detectors as the test set and consistently improves their generalizability as the training set. Additionally, we demonstrate the superiority of our semantics-oriented method over traditional binary and multi-class classification-based detectors.