On the Vulnerability of Data Points under Multiple Membership Inference Attacks and Target Models

TL;DR

This paper investigates the vulnerability of data points to multiple membership inference attacks across various models, introducing new metrics and a platform to analyze and identify vulnerable data points, revealing that vulnerability depends on the attack and model, not the data itself.

Contribution

The paper proposes new metrics to assess data point vulnerability under multiple MIAs and models, and develops a scalable platform for analysis, highlighting the limitations of previous methods.

Findings

MIA has a tendency to infer some data points as vulnerable despite low overall accuracy.

Existing methods are inadequate for identifying vulnerabilities across multiple MIAs and models.

Vulnerability is linked to the attack and model, not inherent data point characteristics.

Abstract

Membership Inference Attacks (MIAs) infer whether a data point is in the training data of a machine learning model. It is a threat while being in the training data is private information of a data point. MIA correctly infers some data points as members or non-members of the training data. Intuitively, data points that MIA accurately detects are vulnerable. Considering those data points may exist in different target models susceptible to multiple MIAs, the vulnerability of data points under multiple MIAs and target models is worth exploring. This paper defines new metrics that can reflect the actual situation of data points' vulnerability and capture vulnerable data points under multiple MIAs and target models. From the analysis, MIA has an inference tendency to some data points despite a low overall inference performance. Additionally, we implement 54 MIAs, whose average attack accuracy ranges from 0.5 to 0.9, to support our analysis with our scalable and flexible platform, Membership Inference Attacks Platform (VMIAP). Furthermore, previous methods are unsuitable for finding vulnerable data points under multiple MIAs and different target models. Finally, we observe that the vulnerability is not characteristic of the data point but related to the MIA and target model.