Spotting tell-tale visual artifacts in face swapping videos: strengths and pitfalls of CNN detectors

TL;DR

This paper evaluates CNN-based detectors for face swapping artifacts, revealing strong performance within datasets but challenges in generalizing across different sources and occlusion scenarios, emphasizing the need for specialized methods.

Contribution

It benchmarks CNN detectors on multiple datasets, including a new one, and analyzes their generalization limits in face swapping artifact detection.

Findings

CNN models perform well within the same dataset

Detection accuracy drops across different data sources

Occlusion-based artifacts are particularly challenging to detect

Abstract

Face swapping manipulations in video streams represents an increasing threat in remote video communications, due to advances in automated and real-time tools. Recent literature proposes to characterize and exploit visual artifacts introduced in video frames by swapping algorithms when dealing with challenging physical scenes, such as face occlusions. This paper investigates the effectiveness of this approach by benchmarking CNN-based data-driven models on two data corpora (including a newly collected one) and analyzing generalization capabilities with respect to different acquisition sources and swapping algorithms. The results confirm excellent performance of general-purpose CNN architectures when operating within the same data source, but a significant difficulty in robustly characterizing occlusion-based visual cues across datasets. This highlights the need for specialized detection strategies to deal with such artifacts.