Revisiting Machine Unlearning with Dimensional Alignment

TL;DR

This paper introduces a novel framework for machine unlearning that uses dimensional alignment to improve stability and effectiveness, addressing flaws in existing evaluation metrics and ensuring models forget specific data while retaining overall knowledge.

Contribution

The paper proposes a new unlearning framework based on eigenspace alignment, a novel evaluation metric called dimensional alignment, and improved training schemes for stable, effective unlearning.

Findings

The proposed method effectively removes information from the forget set.

It preserves knowledge from the retain set.

New evaluation tools better reflect unlearning goals.

Abstract

Machine unlearning, an emerging research topic focusing on compliance with data privacy regulations, enables trained models to remove the information learned from specific data. While many existing methods indirectly address this issue by intentionally injecting incorrect supervisions, they can drastically and unpredictably alter the decision boundaries and feature spaces, leading to training instability and undesired side effects. To fundamentally approach this task, we first analyze the changes in latent feature spaces between original and retrained models, and observe that the feature representations of samples not involved in training are closely aligned with the feature manifolds of previously seen samples in training. Based on these findings, we introduce a novel evaluation metric for machine unlearning, coined dimensional alignment, which measures the alignment between the eigenspaces of the forget and retain set samples. We employ this metric as a regularizer loss to build a robust and stable unlearning framework, which is further enhanced by integrating a self-distillation loss and an alternating training scheme. Our framework effectively eliminates information from the forget set and preserves knowledge from the retain set. Lastly, we identify critical flaws in established evaluation metrics for machine unlearning, and introduce new evaluation tools that more accurately reflect the fundamental goals of machine unlearning.