MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models

TL;DR

MultiPriv introduces a comprehensive benchmark to evaluate how well vision-language models can infer and link personal information, revealing significant privacy risks in current models.

Contribution

We present MultiPriv, the first benchmark for assessing individual-level privacy reasoning in vision-language models, including a new dataset and evaluation framework.

Findings

60% of tested VLMs can perform privacy reasoning with up to 80% accuracy

Existing benchmarks overlook the inference capabilities of VLMs in privacy contexts

MultiPriv highlights the privacy risks posed by current VLMs

Abstract

Modern Vision-Language Models (VLMs) pose significant individual-level privacy risks by linking fragmented multimodal data to identifiable individuals through hierarchical chain-of-thought reasoning. However, existing privacy benchmarks remain structurally insufficient for this threat, as they primarily evaluate privacy perception while failing to address the more critical risk of privacy reasoning: a VLM's ability to infer and link distributed information to construct individual profiles. To address this gap, we propose MultiPriv, the first benchmark designed to systematically evaluate individual-level privacy reasoning in VLMs. We introduce the Privacy Perception and Reasoning (PPR) framework and construct a bilingual multimodal dataset with synthetic individual profiles, where identifiers (e.g., faces, names) are linked to sensitive attributes. This design enables nine challenging tasks spanning attribute detection, cross-image re-identification, and chained inference. We conduct a large-scale evaluation of over 50 open-source and commercial VLMs. Our analysis shows that 60 percent of widely used VLMs can perform individual-level privacy reasoning with up to 80 percent accuracy, posing a significant threat to personal privacy. MultiPriv provides a foundation for developing and assessing privacy-preserving VLMs.