Light as Deception: GPT-driven Natural Relighting Against Vision-Language Pre-training Models

TL;DR

LightD introduces a novel semantically guided relighting framework using ChatGPT and a pretrained relighting model to generate natural adversarial samples for vision-language models, improving attack effectiveness and realism.

Contribution

The paper presents LightD, a new method combining ChatGPT and relighting models to craft natural adversarial samples for VLP models, expanding attack space while preserving scene semantics.

Findings

LightD outperforms existing attack methods in effectiveness.

Generated adversarial samples maintain high visual naturalness.

Demonstrated across multiple VLP tasks and models.

Abstract

While adversarial attacks on vision-and-language pretraining (VLP) models have been explored, generating natural adversarial samples crafted through realistic and semantically meaningful perturbations remains an open challenge. Existing methods, primarily designed for classification tasks, struggle when adapted to VLP models due to their restricted optimization spaces, leading to ineffective attacks or unnatural artifacts. To address this, we propose \textbf{LightD}, a novel framework that generates natural adversarial samples for VLP models via semantically guided relighting. Specifically, LightD leverages ChatGPT to propose context-aware initial lighting parameters and integrates a pretrained relighting model (IC-light) to enable diverse lighting adjustments. LightD expands the optimization space while ensuring perturbations align with scene semantics. Additionally, gradient-based optimization is applied to the reference lighting image to further enhance attack effectiveness while maintaining visual naturalness. The effectiveness and superiority of the proposed LightD have been demonstrated across various VLP models in tasks such as image captioning and visual question answering.