Gone but Not Forgotten: Improved Benchmarks for Machine Unlearning

TL;DR

This paper proposes improved benchmarks and evaluation methods for machine unlearning algorithms, highlighting their privacy and performance trade-offs through comprehensive experiments on computer vision datasets.

Contribution

It introduces alternative evaluation techniques for machine unlearning, addressing limitations of previous assessments and providing a more realistic understanding of privacy and utility.

Findings

Current evaluations are insufficient for realistic privacy threats

Proposed benchmarks reveal nuanced performance and privacy trade-offs

State-of-the-art algorithms show varied effectiveness across datasets

Abstract

Machine learning models are vulnerable to adversarial attacks, including attacks that leak information about the model's training data. There has recently been an increase in interest about how to best address privacy concerns, especially in the presence of data-removal requests. Machine unlearning algorithms aim to efficiently update trained models to comply with data deletion requests while maintaining performance and without having to resort to retraining the model from scratch, a costly endeavor. Several algorithms in the machine unlearning literature demonstrate some level of privacy gains, but they are often evaluated only on rudimentary membership inference attacks, which do not represent realistic threats. In this paper we describe and propose alternative evaluation methods for three key shortcomings in the current evaluation of unlearning algorithms. We show the utility of our alternative evaluations via a series of experiments of state-of-the-art unlearning algorithms on different computer vision datasets, presenting a more detailed picture of the state of the field.