On the Reliability of Vision-Language Models Under Adversarial Frequency-Domain Perturbations

TL;DR

This paper reveals that vision-language models are vulnerable to subtle frequency-domain perturbations, which can significantly affect their performance in image captioning and DeepFake detection, raising concerns about their reliability.

Contribution

The study introduces a systematic method to perturb images in the frequency domain, exposing vulnerabilities of state-of-the-art VLMs across multiple datasets and models.

Findings

Frequency perturbations undermine VLM accuracy.

VLM judgments are sensitive to frequency cues.

Vulnerabilities exist under black-box conditions.

Abstract

Vision-Language Models (VLMs) are increasingly used as perceptual modules for visual content reasoning, including through captioning and DeepFake detection. In this work, we expose a critical vulnerability of VLMs when exposed to subtle, structured perturbations in the frequency domain. Specifically, we highlight how these feature transformations undermine authenticity/DeepFake detection and automated image captioning tasks. We design targeted image transformations, operating in the frequency domain to systematically adjust VLM outputs when exposed to frequency-perturbed real and synthetic images. We demonstrate that the perturbation injection method generalizes across five state-of-the-art VLMs which includes different-parameter Qwen2/2.5 and BLIP models. Experimenting across ten real and generated image datasets reveals that VLM judgments are sensitive to frequency-based cues and may not wholly align with semantic content. Crucially, we show that visually-imperceptible spatial frequency transformations expose the fragility of VLMs deployed for automated image captioning and authenticity detection tasks. Our findings under realistic, black-box constraints challenge the reliability of VLMs, underscoring the need for robust multimodal perception systems.