Learning on Less: Constraining Pre-trained Model Learning for Generalizable Diffusion-Generated Image Detection

TL;DR

This paper introduces Learning on Less (LoL), a training method that constrains pre-trained models to improve the detection of diffusion-generated images, especially from unseen models, with minimal training data.

Contribution

The paper proposes LoL, a novel training approach that leverages pre-trained models and random masking to enhance generalization in diffusion image detection.

Findings

LoL achieves a 13.6% higher accuracy than state-of-the-art with only 1% training data.

Pre-trained models effectively cluster real image features but are unstable in generalization.

LoL focuses on less image content to improve universal feature extraction for diffusion image detection.

Abstract

Diffusion Models enable realistic image generation, raising the risk of misinformation and eroding public trust. Currently, detecting images generated by unseen diffusion models remains challenging due to the limited generalization capabilities of existing methods. To address this issue, we rethink the effectiveness of pre-trained models trained on large-scale, real-world images. Our findings indicate that: 1) Pre-trained models can cluster the features of real images effectively. 2) Models with pre-trained weights can approximate an optimal generalization solution at a specific training step, but it is extremely unstable. Based on these facts, we propose a simple yet effective training method called Learning on Less (LoL). LoL utilizes a random masking mechanism to constrain the model's learning of the unique patterns specific to a certain type of diffusion model, allowing it to focus on less image content. This leverages the inherent strengths of pre-trained weights while enabling a more stable approach to optimal generalization, which results in the extraction of a universal feature that differentiates various diffusion-generated images from real images. Extensive experiments on the GenImage benchmark demonstrate the remarkable generalization capability of our proposed LoL. With just 1% training data, LoL significantly outperforms the current state-of-the-art, achieving a 13.6% improvement in average ACC across images generated by eight different models.