Cross-Forgery Analysis of Vision Transformers and CNNs for Deepfake Image Detection

TL;DR

This study compares Vision Transformers and CNNs in deepfake detection, finding that Transformers generalize better to new manipulation methods, while CNNs perform better on known techniques.

Contribution

It provides a comparative analysis of Vision Transformers and CNNs for deepfake detection, highlighting the superior generalization ability of Transformers in cross-forgery scenarios.

Findings

Vision Transformers outperform CNNs in generalizing to new deepfake methods.

EfficientNetV2 performs better on known deepfake generation techniques.

Transformers show greater adaptability to unseen deepfake generation methods.

Abstract

Deepfake Generation Techniques are evolving at a rapid pace, making it possible to create realistic manipulated images and videos and endangering the serenity of modern society. The continual emergence of new and varied techniques brings with it a further problem to be faced, namely the ability of deepfake detection models to update themselves promptly in order to be able to identify manipulations carried out using even the most recent methods. This is an extremely complex problem to solve, as training a model requires large amounts of data, which are difficult to obtain if the deepfake generation method is too recent. Moreover, continuously retraining a network would be unfeasible. In this paper, we ask ourselves if, among the various deep learning techniques, there is one that is able to generalise the concept of deepfake to such an extent that it does not remain tied to one or more specific deepfake generation methods used in the training set. We compared a Vision Transformer with an EfficientNetV2 on a cross-forgery context based on the ForgeryNet dataset. From our experiments, It emerges that EfficientNetV2 has a greater tendency to specialize often obtaining better results on training methods while Vision Transformers exhibit a superior generalization ability that makes them more competent even on images generated with new methodologies.