Detect Fake with Fake: Leveraging Synthetic Data-driven Representation for Synthetic Image Detection

TL;DR

This paper demonstrates that vision transformers trained solely on synthetic data can effectively detect fake images, outperforming models trained on real data, with significant improvements on unseen GAN models.

Contribution

It shows synthetic data-driven representations, especially with vision transformers, are effective for fake image detection without using real images during training.

Findings

Vision transformers trained on synthetic data can distinguish fake images from real ones.

Using SynCLR backbone improves detection performance over CLIP.

The method achieves +10.32 mAP and +4.73% accuracy on unseen GAN models.

Abstract

Are general-purpose visual representations acquired solely from synthetic data useful for detecting fake images? In this work, we show the effectiveness of synthetic data-driven representations for synthetic image detection. Upon analysis, we find that vision transformers trained by the latest visual representation learners with synthetic data can effectively distinguish fake from real images without seeing any real images during pre-training. Notably, using SynCLR as the backbone in a state-of-the-art detection method demonstrates a performance improvement of +10.32 mAP and +4.73% accuracy over the widely used CLIP, when tested on previously unseen GAN models. Code is available at https://github.com/cvpaperchallenge/detect-fake-with-fake.