Multi-PA: A Multi-perspective Benchmark on Privacy Assessment for Large Vision-Language Models

TL;DR

This paper introduces Multi-PA, a comprehensive benchmark for evaluating privacy risks in large vision-language models, addressing gaps in privacy assessment across multiple categories and dimensions.

Contribution

We propose Multi-PA, the first extensive benchmark covering privacy awareness and leakage across diverse privacy categories for LVLMs.

Findings

Current LVLMs show high privacy breach risks.

Vulnerabilities vary across privacy categories.

Benchmark covers over 31,000 samples.

Abstract

Large Vision-Language Models (LVLMs) exhibit impressive potential across various tasks but also face significant privacy risks, limiting their practical applications. Current researches on privacy assessment for LVLMs is limited in scope, with gaps in both assessment dimensions and privacy categories. To bridge this gap, we propose Multi-PA, a comprehensive benchmark for evaluating the privacy preservation capabilities of LVLMs in terms of privacy awareness and leakage. Privacy awareness measures the model's ability to recognize the privacy sensitivity of input data, while privacy leakage assesses the risk of the model unintentionally disclosing privacy information in its output. We design a range of sub-tasks to thoroughly evaluate the model's privacy protection offered by LVLMs. Multi-PA covers 26 categories of personal privacy, 15 categories of trade secrets, and 18 categories of state secrets, totaling 31,962 samples. Based on Multi-PA, we evaluate the privacy preservation capabilities of 21 open-source and 2 closed-source LVLMs. Our results reveal that current LVLMs generally pose a high risk of facilitating privacy breaches, with vulnerabilities varying across personal privacy, trade secret, and state secret.