Findings of the Counter Turing Test: AI-Generated Image Detection

TL;DR

This paper reports on the Counter Turing Test for detecting AI-generated images, showing high accuracy in distinguishing synthetic from real images but difficulty in identifying the specific generative model.

Contribution

Introduced the CT2 competition and MS COCOAI dataset, advancing methods for AI-generated image detection and model fingerprinting.

Findings

AI-generated images can be detected with high accuracy (F1 > 0.83)

Identifying the specific generative model remains challenging (F1 ~ 0.50)

Diverse detection strategies were employed, including CNNs, ViTs, and frequency analysis.

Abstract

The rapid advancements in generative AI technologies, such as Stable Diffusion, DALL-E, and Midjourney, have significantly transformed the creation of synthetic visual content. While these models enable innovation across industries, they also pose serious challenges, including misinformation, disinformation, and biased content generation. The increasing realism of AI-generated images makes their detection a pressing concern for researchers, policymakers, and industry stakeholders. In this paper, we present the findings of the Defactify 4.0 workshop, which introduced the Counter Turing Test (CT2) for AI-Generated Image Detection. The competition consisted of two key tasks: (1) binary classification of images as either AI-generated or real and (2) identification of the specific generative model responsible for an AI-generated image. To facilitate this, we developed the MS COCOAI dataset, consisting of 50,000 synthetic images from multiple generative models alongside real-world images from the MS COCO dataset. Participants employed diverse detection strategies, including convolutional neural networks (CNNs), Vision Transformers (ViTs), frequency-based analysis, contrastive learning, and multimodal techniques. The results demonstrated that while AI-generated images can be detected with high accuracy (F1-score > 0.83), identifying the exact model used remains significantly more challenging (highest F1-score: 0.4986). These findings highlight the need for improved model fingerprinting, adversarial robustness, and real-time detection mechanisms.