When Visual Privacy Protection Meets Multimodal Large Language Models

TL;DR

This paper proposes a novel framework to protect visual privacy in multimodal large language models deployed as black-box cloud services, balancing privacy and model performance.

Contribution

It introduces a new privacy protection method for black-box MLLMs using Pareto optimality and critical-history optimization techniques.

Findings

Effective privacy protection demonstrated on multiple benchmarks.

Achieves a better trade-off between privacy and MLLM performance.

Applicable to real-world cloud-based MLLM services.

Abstract

The emergence of Multimodal Large Language Models (MLLMs) and the widespread usage of MLLM cloud services such as GPT-4V raised great concerns about privacy leakage in visual data. As these models are typically deployed in cloud services, users are required to submit their images and videos, posing serious privacy risks. However, how to tackle such privacy concerns is an under-explored problem. Thus, in this paper, we aim to conduct a new investigation to protect visual privacy when enjoying the convenience brought by MLLM services. We address the practical case where the MLLM is a "black box", i.e., we only have access to its input and output without knowing its internal model information. To tackle such a challenging yet demanding problem, we propose a novel framework, in which we carefully design the learning objective with Pareto optimality to seek a better trade-off between visual privacy and MLLM's performance, and propose critical-history enhanced optimization to effectively optimize the framework with the black-box MLLM. Our experiments show that our method is effective on different benchmarks.