Sharpness-Aware Machine Unlearning
Haoran Tang, Rajiv Khanna
TL;DR
This paper investigates the effects of Sharpness-Aware Minimization (SAM) on machine unlearning, revealing its limitations and proposing a new method, Sharp MinMax, to improve unlearning performance and model robustness.
Contribution
It provides a refined understanding of SAM's behavior in unlearning, introduces Sharp MinMax for better signal separation, and demonstrates improved unlearning and robustness across various settings.
Findings
SAM outperforms SGD in unlearning tasks.
SAM reduces feature entanglement and enhances resistance to membership inference.
Sharp MinMax achieves superior unlearning performance by splitting model training.
Abstract
We characterize the effectiveness of Sharpness-aware minimization (SAM) under machine unlearning scheme, where unlearning forget signals interferes with learning retain signals. While previous work prove that SAM improves generalization with noise memorization prevention, we show that SAM abandons such denoising property when fitting the forget set, leading to altered generalization depending on signal strength. We further characterize the signal surplus of SAM in the order of signal strength, which enables learning from less retain signals to maintain model performance and putting more weight on unlearning the forget set. Empirical studies show that SAM outperforms SGD with relaxed requirement for retain signals and can enhance various unlearning methods either as pretrain or unlearn algorithm. Motivated by our refined characterization of SAM unlearning and observing that overfitting…
Peer Reviews
Decision·ICLR 2026 Poster
Theoretical analysis of SAM is robust. The paper has done a good job of analyzing it in relation to Random Labeling and especially NegGrad. I appreciate they were honest about SAM+NegGrad giving worse forget accuracy than SGD+NegGrad before their discussion of overfitting.
1. UMAP Visualization Clarity The UMAP visualizations are difficult to interpret. For instance, Figure 1 is intended to illustrate inter- and intra-class movements after unlearning; however, these differences are not visually discernible. The colors used are too similar, and the expected variations are not immediately perceptible. I recommend improving visual clarity by adopting a more distinct color palette, varying marker shapes, or explicitly highlighting changes using arrows, circles, or oth
1. The paper provide in-depth theoretical analysis on how SAM would impact the unlearning quality. 2. Experiments contains sufficient evidence on how well SAM argumented model performs compared to the original classic unlearning methods.
1. The paper is not well polished and contains numerous grammatical errors and excessive notation, which makes it difficult to read. While analytical papers often introduce complex notation to convey ideas precisely, this paper’s presentation suffers from a combination of grammar issues, overly long sentences, and vague explanations. A few specific examples: 1. Equation 8 and corresponding description is very vague, and I cannot get how this equation is used in the proposed work. Is it on fo
1. The authors develop a signal–noise decomposition framework to analyze how SAM and SGD behave when learning (retain) and unlearning (forget) signals interact, giving novel insights on how SAM affects unlearning performance. 2. The paper further proposes Sharpness MinMax, a novel optimization method that improves machine unlearning performance. 3. The effectiveness of the proposed method is evaluated on multiple datasets. The paper also gives additional evaluations such as feature entanglement
1. For the evaluation metrics, run-time evaluations are missing. Employing SAM on both forget and retain loss might significantly increase run time, and this efficiency trade-off should be evaluated. 2. Evaluations of unlearning robustness are missing, such as evaluating the performance of the unlearned models against relearning attacks [1]. Sharpness-aware minimization employed in LLM unlearning [2] increases the robustness of the unlearned models against relearning by perturbing the model wit
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Stream Mining Techniques · Online Learning and Analytics · Machine Learning and Data Classification
MethodsSegment Anything Model · Sharpness-Aware Minimization · Stochastic Gradient Descent