EMA: Auditing Data Removal from Trained Models

TL;DR

This paper introduces EMA, a robust method for auditing whether specific data has been removed from trained models, outperforming previous approaches under various practical conditions.

Contribution

We propose Ensembled Membership Auditing (EMA), a novel method that overcomes limitations of prior data removal auditing techniques in neural networks.

Findings

EMA is robust under various conditions.

EMA outperforms previous methods in failure cases.

Experiments on MNIST, SVHN, and Chest X-ray datasets confirm effectiveness.

Abstract

Data auditing is a process to verify whether certain data have been removed from a trained model. A recently proposed method (Liu et al. 20) uses Kolmogorov-Smirnov (KS) distance for such data auditing. However, it fails under certain practical conditions. In this paper, we propose a new method called Ensembled Membership Auditing (EMA) for auditing data removal to overcome these limitations. We compare both methods using benchmark datasets (MNIST and SVHN) and Chest X-ray datasets with multi-layer perceptrons (MLP) and convolutional neural networks (CNN). Our experiments show that EMA is robust under various conditions, including the failure cases of the previously proposed method. Our code is available at: https://github.com/Hazelsuko07/EMA.