On Newton's Method to Unlearn Neural Networks

TL;DR

This paper introduces CureNewton, a novel cubic regularization-based method for approximate unlearning in neural networks, addressing Hessian degeneracy issues and achieving competitive performance efficiently.

Contribution

It proposes CureNewton, a new unlearning algorithm that effectively handles Hessian degeneracy using cubic regularization, improving upon existing methods.

Findings

CureNewton achieves unlearning performance comparable to state-of-the-art methods.

The method is theoretically justified and computationally efficient.

Experiments demonstrate effectiveness across various models and datasets.

Abstract

With the widespread applications of neural networks (NNs) trained on personal data, machine unlearning has become increasingly important for enabling individuals to exercise their personal data ownership, particularly the "right to be forgotten" from trained NNs. Since retraining is computationally expensive, we seek approximate unlearning algorithms for NNs that return identical models to the retrained oracle. While Newton's method has been successfully used to approximately unlearn linear models, we observe that adapting it for NN is challenging due to degenerate Hessians that make computing Newton's update impossible. Additionally, we show that when coupled with popular techniques to resolve the degeneracy, Newton's method often incurs offensively large norm updates and empirically degrades model performance post-unlearning. To address these challenges, we propose CureNewton's method, a principle approach that leverages cubic regularization to handle the Hessian degeneracy effectively. The added regularizer eliminates the need for manual finetuning and affords a natural interpretation within the unlearning context. Experiments across different models and datasets show that our method can achieve competitive unlearning performance to the state-of-the-art algorithm in practical unlearning settings, while being theoretically justified and efficient in running time.