Backbone is All You Need: Assessing Vulnerabilities of Frozen Foundation Models in Synthetic Image Forensics

TL;DR

This paper introduces SIAA, a gray-box attack exploiting ViT backbone knowledge to generate highly effective adversarial examples against synthetic image detectors, revealing a critical vulnerability.

Contribution

We demonstrate that knowledge of the ViT backbone alone enables highly successful adversarial attacks, exposing a key vulnerability in frozen foundation models for image forensics.

Findings

High attack success rates approaching white-box performance.

Backbone knowledge alone suffices to undermine detector reliability.

Vulnerability persists across multiple ViT-based detectors and scenarios.

Abstract

As AI-generated synthetic images become increasingly realistic, Vision Transformers (ViTs) have emerged as a cornerstone of modern deepfake detection. However, the prevailing reliance on frozen, pre-trained backbones introduces a subtle yet critical vulnerability. In this work, we present the Surrogate Iterative Adversarial Attack (SIAA), a gray-box attack that exploits knowledge of the detector's ViT backbone alone and operates entirely within the target detector's feature space to craft highly effective adversarial examples. Through our experiments, involving multiple ViT-based detectors and diverse gray-box scenarios, including few-shot learning, complete training misalignment and attack transferability tests, we demonstrate that this vulnerability consistently yields high attack success rates, often approaching white-box performance. By doing so, we reveal that backbone knowledge alone is sufficient to undermine detector reliability, highlighting the urgent need for more resilient defenses in adversarial multimedia forensics.