Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data

TL;DR

This paper argues that membership inference attacks cannot reliably prove a model was trained on specific data due to fundamental limitations, and suggests alternative methods like data extraction attacks for valid proofs.

Contribution

The paper critically analyzes the limitations of membership inference attacks for training data proofs and proposes alternative approaches such as data extraction attacks.

Findings

Membership inference attacks cannot reliably prove training data inclusion.

Sampling from the null hypothesis is infeasible without knowing the training set.

Data extraction and canary data attacks can provide sound proofs.

Abstract

We consider the problem of a training data proof, where a data creator or owner wants to demonstrate to a third party that some machine learning model was trained on their data. Training data proofs play a key role in recent lawsuits against foundation models trained on web-scale data. Many prior works suggest to instantiate training data proofs using membership inference attacks. We argue that this approach is fundamentally unsound: to provide convincing evidence, the data creator needs to demonstrate that their attack has a low false positive rate, i.e., that the attack's output is unlikely under the null hypothesis that the model was not trained on the target data. Yet, sampling from this null hypothesis is impossible, as we do not know the exact contents of the training set, nor can we (efficiently) retrain a large foundation model. We conclude by offering two paths forward, by showing that data extraction attacks and membership inference on special canary data can be used to create sound training data proofs.