Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Models

TL;DR

This paper introduces a novel membership inference attack method for large vision-language models, leveraging their different responses to image corruption to determine if an image was part of the training data.

Contribution

The paper proposes ICIMIA, a simple yet effective attack method inspired by image corruption sensitivity, applicable in both white-box and black-box scenarios for LVLMs.

Findings

ICIMIA achieves high accuracy in membership inference.

The method is effective under both white-box and black-box settings.

Experiments validate the attack's effectiveness on multiple datasets.

Abstract

Large vision-language models (LVLMs) have demonstrated outstanding performance in many downstream tasks. However, LVLMs are trained on large-scale datasets, which can pose privacy risks if training images contain sensitive information. Therefore, it is important to detect whether an image is used to train the LVLM. Recent studies have investigated membership inference attacks (MIAs) against LVLMs, including detecting image-text pairs and single-modality content. In this work, we focus on detecting whether a target image is used to train the target LVLM. We design simple yet effective Image Corruption-Inspired Membership Inference Attacks (ICIMIA) against LVLMs, which are inspired by LVLM's different sensitivity to image corruption for member and non-member images. We first perform an MIA method under the white-box setting, where we can obtain the embeddings of the image through the vision part of the target LVLM. The attacks are based on the embedding similarity between the image and its corrupted version. We further explore a more practical scenario where we have no knowledge about target LVLMs and we can only query the target LVLMs with an image and a textual instruction. We then conduct the attack by utilizing the output text embeddings' similarity. Experiments on existing datasets validate the effectiveness of our proposed methods under those two different settings.