LEGO: LoRA-Enabled Generator-Oriented Framework for Synthetic Image Detection

TL;DR

LEGO is a modular framework that improves synthetic image detection by capturing generator-specific artifacts using specialized LoRA modules, enhancing generalization with limited data.

Contribution

LEGO introduces a generator-oriented approach with multiple LoRA modules and attention fusion, outperforming prior methods with less training data and fewer epochs.

Findings

LEGO achieves superior detection accuracy over state-of-the-art methods.

It requires less training data and fewer epochs to adapt to new generators.

The modular design allows easy extension to new datasets.

Abstract

The rapid advancement of generative technologies has made synthetic images nearly indistinguishable from real ones, thereby creating an urgent need for robust detectors to counter misinformation. However, existing methods mainly rely on universal artifact features that are shared across multiple generators. We observe that as the diversity of generators increases, the overlap of these common features gradually decreases. This severely undermines model generalization. In contrast, focusing only on unique artifacts tends to cause overfitting to specific forgery patterns. To address this challenge, we propose LEGO (LoRA-Enabled Generator-Oriented Framework). The core mechanism of LEGO employs an MLP to modulate multiple LoRA (Low-Rank Adaptation) blocks, each pretrained to capture the unique artifacts of a specific generator, followed by attention-based feature fusion. Unlike conventional methods that seek a single universal solution, LEGO delegates unique artifact extraction to specialized LoRA modules by dividing its training procedure into two stages. Each LoRA module is individually trained on a single-generator dataset to learn generator-specific representations, then MLP and attention layers are trained on mixed datasets to dynamically regulate the contribution of each module. Benefiting from its modular yet robust design, LEGO can be naturally extended by incorporating new LoRA modules for adaptation to newly emerging next-generation datasets, while still achieving substantially better performance than prior SOTA methods with fewer than 30,000 training images, less than 10% of their training data, and only 5 epochs in each training stage.