Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers

TL;DR

This paper examines how second-order optimizers handle data deletion in machine unlearning, revealing residual information and stability issues not present in first-order methods.

Contribution

It highlights the limitations of current unlearning definitions for second-order optimizers and analyzes the geometric aspects affecting their memory and stability.

Findings

Second-order optimizers exhibit residual information after unlearning.

Stability and information loss are restored only with controlled state perturbation.

Both first and second-order methods align with ideal counterfactual performance after unlearning.

Abstract

We argue that current definitions of machine unlearning are underspecified for second-order optimizers. We compare first-order and second-order learners for their ability to handle the data deletion task with varying degrees of eigendecomposition to mimic the loss model memory. While both first and second-order methods realign with the ideal counterfactul in terms of performance and gradient, the second-order optimizer shows significant volatility in the optimizer state. This indicates residual information, supposedly deleted, that isn't detectable by first-order analysis. Various eigendecay treatments show that stability and information loss is regained only under controlled state pertubation where geometric information (or memory) is erased.