A Sanity Check for AI-generated Image Detection

TL;DR

This paper evaluates the effectiveness of existing AI-generated image detectors, introduces a new challenging dataset called Chameleon, and proposes AIDE, a hybrid feature-based detector that improves detection accuracy but the problem remains unsolved.

Contribution

The paper presents Chameleon, a challenging dataset for AI-generated image detection, and introduces AIDE, a hybrid model combining high-level semantics and low-level artifacts for improved detection.

Findings

Most existing detectors fail on the Chameleon dataset.

AIDE outperforms state-of-the-art methods by 3.5% and 4.6%.

AI-generated image detection remains a challenging problem.

Abstract

With the rapid development of generative models, discerning AI-generated content has evoked increasing attention from both industry and academia. In this paper, we conduct a sanity check on "whether the task of AI-generated image detection has been solved". To start with, we present Chameleon dataset, consisting AIgenerated images that are genuinely challenging for human perception. To quantify the generalization of existing methods, we evaluate 9 off-the-shelf AI-generated image detectors on Chameleon dataset. Upon analysis, almost all models classify AI-generated images as real ones. Later, we propose AIDE (AI-generated Image DEtector with Hybrid Features), which leverages multiple experts to simultaneously extract visual artifacts and noise patterns. Specifically, to capture the high-level semantics, we utilize CLIP to compute the visual embedding. This effectively enables the model to discern AI-generated images based on semantics or contextual information; Secondly, we select the highest frequency patches and the lowest frequency patches in the image, and compute the low-level patchwise features, aiming to detect AI-generated images by low-level artifacts, for example, noise pattern, anti-aliasing, etc. While evaluating on existing benchmarks, for example, AIGCDetectBenchmark and GenImage, AIDE achieves +3.5% and +4.6% improvements to state-of-the-art methods, and on our proposed challenging Chameleon benchmarks, it also achieves the promising results, despite this problem for detecting AI-generated images is far from being solved.