VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models

TL;DR

This paper introduces a novel adversarial attack method that selectively conceals sensitive regions in images to protect user privacy in vision-language models, maintaining overall image semantics.

Contribution

It presents a new targeted attack strategy that preserves image coherence while effectively hiding private information from state-of-the-art VLMs.

Findings

Achieves up to 98% reduction in ROI detection

Maintains high semantic similarity between original and adversarial images

Works across multiple advanced VLMs

Abstract

Recent years have witnessed remarkable progress in developing Vision-Language Models (VLMs) capable of processing both textual and visual inputs. These models have demonstrated impressive performance, leading to their widespread adoption in various applications. However, this widespread raises serious concerns regarding user privacy, particularly when models inadvertently process or expose private visual information. In this work, we frame the preservation of privacy in VLMs as an adversarial attack problem. We propose a novel attack strategy that selectively conceals information within designated Region Of Interests (ROIs) in an image, effectively preventing VLMs from accessing sensitive content while preserving the semantic integrity of the remaining image. Unlike conventional adversarial attacks that often disrupt the entire image, our method maintains high coherence in unmasked areas. Experimental results across three state-of-the-art VLMs namely LLaVA, Instruct-BLIP, and BLIP2-T5 demonstrate up to 98% reduction in detecting targeted ROIs, while maintaining global image semantics intact, as confirmed by high similarity scores between clean and adversarial outputs. We believe that this work contributes to a more privacy conscious use of multimodal models and offers a practical tool for further research, with the source code publicly available at: https://github.com/hbrachemi/Vlm_defense-attack.