Leveraging Hierarchical Image-Text Misalignment for Universal Fake Image Detection

TL;DR

This paper introduces ITEM, a novel fake image detection method that leverages hierarchical image-text misalignment in a joint visual-language space, improving generalization over existing visual-only approaches.

Contribution

The paper proposes a multi-modal detection approach using hierarchical image-text misalignment in CLIP space, enhancing robustness and generalization in fake image detection.

Findings

Outperforms state-of-the-art methods in generalization

Effective in detecting images from unseen generative models

Robust against various image manipulations

Abstract

With the rapid development of generative models, detecting generated fake images to prevent their malicious use has become a critical issue recently. Existing methods frame this challenge as a naive binary image classification task. However, such methods focus only on visual clues, yielding trained detectors susceptible to overfitting specific image patterns and incapable of generalizing to unseen models. In this paper, we address this issue from a multi-modal perspective and find that fake images cannot be properly aligned with corresponding captions compared to real images. Upon this observation, we propose a simple yet effective detector termed ITEM by leveraging the image-text misalignment in a joint visual-language space as discriminative clues. Specifically, we first measure the misalignment of the images and captions in pre-trained CLIP's space, and then tune a MLP head to perform the usual detection task. Furthermore, we propose a hierarchical misalignment scheme that first focuses on the whole image and then each semantic object described in the caption, which can explore both global and fine-grained local semantic misalignment as clues. Extensive experiments demonstrate the superiority of our method against other state-of-the-art competitors with impressive generalization and robustness on various recent generative models.