Membership Inference Attacks Against Vision-Language Models

TL;DR

This paper investigates the privacy risks of vision-language models by developing novel membership inference attacks that can accurately detect whether specific data was used in training, highlighting potential data leakage issues.

Contribution

It introduces four new membership inference methods tailored for vision-language models, addressing limitations of existing techniques and focusing on sensitive instruction tuning data.

Findings

Achieves over 0.8 AUC in membership inference on small sample sets

Demonstrates vulnerability of VLMs to membership inference attacks

Provides a comprehensive evaluation of attack effectiveness

Abstract

Vision-Language Models (VLMs), built on pre-trained vision encoders and large language models (LLMs), have shown exceptional multi-modal understanding and dialog capabilities, positioning them as catalysts for the next technological revolution. However, while most VLM research focuses on enhancing multi-modal interaction, the risks of data misuse and leakage have been largely unexplored. This prompts the need for a comprehensive investigation of such risks in VLMs. In this paper, we conduct the first analysis of misuse and leakage detection in VLMs through the lens of membership inference attack (MIA). In specific, we focus on the instruction tuning data of VLMs, which is more likely to contain sensitive or unauthorized information. To address the limitation of existing MIA methods, we introduce a novel approach that infers membership based on a set of samples and their sensitivity to temperature, a unique parameter in VLMs. Based on this, we propose four membership inference methods, each tailored to different levels of background knowledge, ultimately arriving at the most challenging scenario. Our comprehensive evaluations show that these methods can accurately determine membership status, e.g., achieving an AUC greater than 0.8 targeting a small set consisting of only 5 samples on LLaVA.