Forgetting Similar Samples: Can Machine Unlearning Do it Better?

TL;DR

This paper critically evaluates existing machine unlearning methods, revealing their limitations in removing influence from similar samples and proposing potential improvements based on comprehensive experiments.

Contribution

It provides a thorough analysis of the effectiveness of current unlearning techniques in the presence of similar samples, highlighting fundamental gaps and suggesting directions for enhancement.

Findings

Most existing methods do not fully remove influence of target samples

Significant performance gap compared to retraining-from-scratch baseline

Identifies limitations in current unlearning strategies for similar samples

Abstract

Machine unlearning, a process enabling pre-trained models to remove the influence of specific training samples, has attracted significant attention in recent years. Although extensive research has focused on developing efficient machine unlearning strategies, we argue that these methods mainly aim at removing samples rather than removing samples' influence on the model, thus overlooking the fundamental definition of machine unlearning. In this paper, we first conduct a comprehensive study to evaluate the effectiveness of existing unlearning schemes when the training dataset includes many samples similar to those targeted for unlearning. Specifically, we evaluate: Do existing unlearning methods truly adhere to the original definition of machine unlearning and effectively eliminate all influence of target samples when similar samples are present in the training dataset? Our extensive experiments, conducted on four carefully constructed datasets with thorough analysis, reveal a notable gap between the expected and actual performance of most existing unlearning methods for image and language models, even for the retraining-from-scratch baseline. Additionally, we also explore potential solutions to enhance current unlearning approaches.