Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models

TL;DR

This paper introduces a novel dataset-level membership inference method based on self-comparison and paraphrasing, which effectively detects training data membership in large vision-language models without needing ground-truth non-member data.

Contribution

The paper proposes a new self-comparison based dataset inference technique that does not require ground-truth non-member data, improving practicality and effectiveness over existing methods.

Findings

Outperforms traditional membership inference attacks and dataset inference techniques.

Effective across various datasets and models, including commercial APIs.

Does not require access to ground-truth non-member data.

Abstract

Large Language Models (LLMs) and Vision-Language Models (VLMs) have made significant advancements in a wide range of natural language processing and vision-language tasks. Access to large web-scale datasets has been a key factor in their success. However, concerns have been raised about the unauthorized use of copyrighted materials and potential copyright infringement. Existing methods, such as sample-level Membership Inference Attacks (MIA) and distribution-based dataset inference, distinguish member data (data used for training) and non-member data by leveraging the common observation that models tend to memorize and show greater confidence in member data. Nevertheless, these methods face challenges when applied to LLMs and VLMs, such as the requirement for ground-truth member data or non-member data that shares the same distribution as the test data. In this paper, we propose a novel dataset-level membership inference method based on Self-Comparison. We find that a member prefix followed by a non-member suffix (paraphrased from a member suffix) can further trigger the model's memorization on training data. Instead of directly comparing member and non-member data, we introduce paraphrasing to the second half of the sequence and evaluate how the likelihood changes before and after paraphrasing. Unlike prior approaches, our method does not require access to ground-truth member data or non-member data in identical distribution, making it more practical. Extensive experiments demonstrate that our proposed method outperforms traditional MIA and dataset inference techniques across various datasets and models, including including public models, fine-tuned models, and API-based commercial models.