Deep Unlearn: Benchmarking Machine Unlearning for Image Classification

TL;DR

This paper provides a comprehensive benchmark of 18 machine unlearning methods for deep neural networks, evaluating their effectiveness and efficiency across multiple datasets, models, and initializations, highlighting the importance of hyperparameter tuning and better baselines.

Contribution

It offers the first extensive evaluation of MU methods on DNNs, comparing their performance with rigorous baselines and emphasizing hyperparameter importance.

Findings

MSG and CT outperform other methods in accuracy and efficiency

Proper hyperparameter tuning is crucial for MU success

NG+ is a strong baseline for future MU evaluations

Abstract

Machine unlearning (MU) aims to remove the influence of particular data points from the learnable parameters of a trained machine learning model. This is a crucial capability in light of data privacy requirements, trustworthiness, and safety in deployed models. MU is particularly challenging for deep neural networks (DNNs), such as convolutional nets or vision transformers, as such DNNs tend to memorize a notable portion of their training dataset. Nevertheless, the community lacks a rigorous and multifaceted study that looks into the success of MU methods for DNNs. In this paper, we investigate 18 state-of-the-art MU methods across various benchmark datasets and models, with each evaluation conducted over 10 different initializations, a comprehensive evaluation involving MU over 100K models. We show that, with the proper hyperparameters, Masked Small Gradients (MSG) and Convolution Transpose (CT), consistently perform better in terms of model accuracy and run-time efficiency across different models, datasets, and initializations, assessed by population-based membership inference attacks (MIA) and per-sample unlearning likelihood ratio attacks (U-LiRA). Furthermore, our benchmark highlights the fact that comparing a MU method only with commonly used baselines, such as Gradient Ascent (GA) or Successive Random Relabeling (SRL), is inadequate, and we need better baselines like Negative Gradient Plus (NG+) with proper hyperparameter selection.