Is Artificial Intelligence Generated Image Detection a Solved Problem?

TL;DR

This paper introduces AIGIBench, a comprehensive benchmark for evaluating the robustness and generalization of AI-generated image detectors in real-world scenarios, revealing significant performance gaps.

Contribution

The paper presents AIGIBench, a new benchmark that rigorously tests state-of-the-art AIGI detectors across diverse real-world challenges and datasets.

Findings

Detectors perform poorly on real-world data compared to controlled settings.

Common data augmentations offer limited improvements in detection robustness.

Pre-processing effects on detection accuracy are nuanced and inconsistent.

Abstract

The rapid advancement of generative models, such as GANs and Diffusion models, has enabled the creation of highly realistic synthetic images, raising serious concerns about misinformation, deepfakes, and copyright infringement. Although numerous Artificial Intelligence Generated Image (AIGI) detectors have been proposed, often reporting high accuracy, their effectiveness in real-world scenarios remains questionable. To bridge this gap, we introduce AIGIBench, a comprehensive benchmark designed to rigorously evaluate the robustness and generalization capabilities of state-of-the-art AIGI detectors. AIGIBench simulates real-world challenges through four core tasks: multi-source generalization, robustness to image degradation, sensitivity to data augmentation, and impact of test-time pre-processing. It includes 23 diverse fake image subsets that span both advanced and widely adopted image generation techniques, along with real-world samples collected from social media and AI art platforms. Extensive experiments on 11 advanced detectors demonstrate that, despite their high reported accuracy in controlled settings, these detectors suffer significant performance drops on real-world data, limited benefits from common augmentations, and nuanced effects of pre-processing, highlighting the need for more robust detection strategies. By providing a unified and realistic evaluation framework, AIGIBench offers valuable insights to guide future research toward dependable and generalizable AIGI detection.Data and code are publicly available at: https://github.com/HorizonTEL/AIGIBench.