DGS-Net: Distillation-Guided Gradient Surgery for CLIP Fine-Tuning in AI-Generated Image Detection

TL;DR

DGS-Net is a novel framework that enhances CLIP fine-tuning for AI-generated image detection by preserving pre-trained knowledge and suppressing irrelevant features through gradient-space decomposition.

Contribution

The paper introduces a gradient-space decomposition method and a distillation-guided approach to improve CLIP fine-tuning for synthetic image detection, reducing catastrophic forgetting.

Findings

Outperforms state-of-the-art methods by an average of 6.6 in detection accuracy.

Demonstrates superior generalization across 50 generative models.

Effectively preserves pre-trained priors while suppressing irrelevant features.

Abstract

The rapid progress of generative models such as GANs and diffusion models has led to the widespread proliferation of AI-generated images, raising concerns about misinformation, privacy violations, and trust erosion in digital media. Although large-scale multimodal models like CLIP offer strong transferable representations for detecting synthetic content, fine-tuning them often induces catastrophic forgetting, which degrades pre-trained priors and limits cross-domain generalization. To address this issue, we propose the Distillation-guided Gradient Surgery Network (DGS-Net), a novel framework that preserves transferable pre-trained priors while suppressing task-irrelevant components. Specifically, we introduce a gradient-space decomposition that separates harmful and beneficial descent directions during optimization. By projecting task gradients onto the orthogonal complement of harmful directions and aligning with beneficial ones distilled from a frozen CLIP encoder, DGS-Net achieves unified optimization of prior preservation and irrelevant suppression. Extensive experiments on 50 generative models demonstrate that our method outperforms state-of-the-art approaches by an average margin of 6.6, achieving superior detection performance and generalization across diverse generation techniques.