Unlearning's Blind Spots: Over-Unlearning and Prototypical Relearning Attack

TL;DR

This paper identifies critical blind spots in machine unlearning, specifically over-unlearning and relearning attacks, and proposes Spotter, a method to mitigate these issues, demonstrating improved robustness on multiple datasets.

Contribution

The paper introduces OU@epsilon to measure over-unlearning, exposes the Prototypical Relearning Attack, and proposes Spotter to counter both issues effectively.

Findings

Spotter reduces over-unlearning damage near forget sets.

Spotter neutralizes Prototypical Relearning Attacks with embedding dispersion.

State-of-the-art results achieved on CIFAR, TinyImageNet, and CASIA-WebFace.

Abstract

Machine unlearning (MU) aims to expunge a designated forget set from a trained model without costly retraining, yet the existing techniques overlook two critical blind spots: "over-unlearning" that deteriorates retained data near the forget set, and post-hoc "relearning" attacks that aim to resurrect the forgotten knowledge. Focusing on class-level unlearning, we first derive an over-unlearning metric, OU@epsilon, which quantifies collateral damage in regions proximal to the forget set, where over-unlearning mainly appears. Next, we expose an unforeseen relearning threat on MU, i.e., the Prototypical Relearning Attack, which exploits the per-class prototype of the forget class with just a few samples, and easily restores the pre-unlearning performance. To counter both blind spots in class-level unlearning, we introduce Spotter, a plug-and-play objective that combines (i) a masked knowledge-distillation penalty on the nearby region of forget classes to suppress OU@epsilon, and (ii) an intra-class dispersion loss that scatters forget-class embeddings, neutralizing Prototypical Relearning Attacks. Spotter achieves state-of-the-art results across CIFAR, TinyImageNet, and CASIA-WebFace datasets, offering a practical remedy to unlearning's blind spots.