PII-VisBench: Evaluating Personally Identifiable Information Safety in Vision Language Models Along a Continuum of Visibility

TL;DR

This paper introduces PII-VisBench, a benchmark for evaluating vision language models' privacy safety across different levels of online presence, revealing model vulnerabilities and the influence of subject visibility on PII leakage.

Contribution

The paper presents a novel benchmark that stratifies subjects by online visibility to evaluate VLM privacy safety, highlighting model heterogeneity and vulnerabilities to prompts.

Findings

Refusal rates increase as subject visibility decreases.

Models are more likely to disclose PII for highly visible subjects.

Prompt variations can expose model vulnerabilities and failures.

Abstract

Vision Language Models (VLMs) are increasingly integrated into privacy-critical domains, yet existing evaluations of personally identifiable information (PII) leakage largely treat privacy as a static extraction task and ignore how a subject's online presence--the volume of their data available online--influences privacy alignment. We introduce PII-VisBench, a novel benchmark containing 4000 unique probes designed to evaluate VLM safety through the continuum of online presence. The benchmark stratifies 200 subjects into four visibility categories: high, medium, low, and zero--based on the extent and nature of their information available online. We evaluate 18 open-source VLMs (0.3B-32B) based on two key metrics: percentage of PII probing queries refused (Refusal Rate) and the fraction of non-refusal responses flagged for containing PII (Conditional PII Disclosure Rate). Across models, we observe a consistent pattern: refusals increase and PII disclosures decrease (9.10% high to 5.34% low) as subject visibility drops. We identify that models are more likely to disclose PII for high-visibility subjects, alongside substantial model-family heterogeneity and PII-type disparities. Finally, paraphrasing and jailbreak-style prompts expose attack and model-dependent failures, motivating visibility-aware safety evaluation and training interventions.