Who Can See Through You? Adversarial Shielding Against VLM-Based Attribute Inference Attacks

TL;DR

This paper introduces a privacy protection method for vision-language models that balances privacy preservation with visual quality, validated on a new benchmark dataset and shown to be effective across multiple models.

Contribution

A novel joint optimization approach for privacy suppression that maintains visual consistency, along with the VPI-COCO benchmark for evaluating privacy methods in VLMs.

Findings

Reduces privacy attribute inference rate (PAR) below 25%

Maintains non-private attribute inference rate (NPAR) above 88%

Generalizes well to unseen and paraphrased privacy questions

Abstract

As vision-language models (VLMs) become widely adopted, VLM-based attribute inference attacks have emerged as a serious privacy concern, enabling adversaries to infer private attributes from images shared on social media. This escalating threat calls for dedicated protection methods to safeguard user privacy. However, existing methods often degrade the visual quality of images or interfere with vision-based functions on social media, thereby failing to achieve a desirable balance between privacy protection and user experience. To address this challenge, we propose a novel protection method that jointly optimizes privacy suppression and utility preservation under a visual consistency constraint. While our method is conceptually effective, fair comparisons between methods remain challenging due to the lack of publicly available evaluation datasets. To fill this gap, we introduce VPI-COCO, a publicly available benchmark comprising 522 images with hierarchically structured privacy questions and corresponding non-private counterparts, enabling fine-grained and joint evaluation of protection methods in terms of privacy preservation and user experience. Building upon this benchmark, experiments on multiple VLMs demonstrate that our method effectively reduces PAR below 25%, keeps NPAR above 88%, maintains high visual consistency, and generalizes well to unseen and paraphrased privacy questions, demonstrating its strong practical applicability for real-world VLM deployments.