When Semantics Regulate: Rethinking Patch Shuffle and Internal Bias for Generated Image Detection with CLIP

TL;DR

This paper reveals that disrupting semantic cues with Patch Shuffle enhances CLIP's ability to detect AI-generated images by focusing on artifacts, leading to improved cross-domain robustness.

Contribution

It uncovers the role of semantic bias in CLIP-based detection and introduces SemAnti, a fine-tuning method that improves generalization by regulating semantics.

Findings

Patch Shuffle reduces semantic entropy and homogenizes feature distributions.

SemAnti achieves state-of-the-art cross-domain detection performance.

Regulating semantics enhances CLIP's robustness for AI-generated image detection.

Abstract

The rapid progress of GANs and Diffusion Models poses new challenges for detecting AI-generated images. Although CLIP-based detectors exhibit promising generalization, they often rely on semantic cues rather than generator artifacts, leading to brittle performance under distribution shifts. In this work, we revisit the nature of semantic bias and uncover that Patch Shuffle provides an unusually strong benefit for CLIP, that disrupts global semantic continuity while preserving local artifact cues, which reduces semantic entropy and homogenizes feature distributions between natural and synthetic images. Through a detailed layer-wise analysis, we further show that CLIP's deep semantic structure functions as a regulator that stabilizes cross-domain representations once semantic bias is suppressed. Guided by these findings, we propose SemAnti, a semantic-antagonistic fine-tuning paradigm that freezes the semantic subspace and adapts only artifact-sensitive layers under shuffled semantics. Despite its simplicity, SemAnti achieves state-of-the-art cross-domain generalization on AIGCDetectBenchmark and GenImage, demonstrating that regulating semantics is key to unlocking CLIP's full potential for robust AI-generated image detection.