Why Some Models Resist Unlearning: A Linear Stability Perspective

TL;DR

This paper introduces a linear stability framework to understand why certain models resist unlearning, linking data geometry, memorization, and optimization dynamics through theoretical analysis and empirical validation.

Contribution

It provides the first theoretical analysis of unlearning stability using asymptotic linear stability and data coherence, connecting memorization levels to unlearning difficulty.

Findings

Models with higher memorization resist unlearning due to increased data coherence.

We identify stability thresholds that distinguish between successful and failed unlearning.

Empirical tests with Hessian and heatmaps validate the theoretical stability boundaries.

Abstract

Machine unlearning, the ability to erase the effect of specific training samples without retraining from scratch, is critical for privacy, regulation, and efficiency. However, most progress in unlearning has been empirical, with little theoretical understanding of when and why unlearning works. We tackle this gap by framing unlearning through the lens of asymptotic linear stability to capture the interaction between optimization dynamics and data geometry. The key quantity in our analysis is data coherence which is the cross sample alignment of loss surface directions near the optimum. We decompose coherence along three axes: within the retain set, within the forget set, and between them, and prove tight stability thresholds that separate convergence from divergence. To further link data properties to forgettability, we study a two layer ReLU CNN under a signal plus noise model and show that stronger memorization makes forgetting easier: when the signal to noise ratio (SNR) is lower, cross sample alignment is weaker, reducing coherence and making unlearning easier; conversely, high SNR, highly aligned models resist unlearning. For empirical verification, we show that Hessian tests and CNN heatmaps align closely with the predicted boundary, mapping the stability frontier of gradient based unlearning as a function of batching, mixing, and data/model alignment. Our analysis is grounded in random matrix theory tools and provides the first principled account of the trade offs between memorization, coherence, and unlearning.