Machine Unlearning under Overparameterization

TL;DR

This paper introduces a new framework for machine unlearning in overparameterized models, defining unlearning as finding the minimum-complexity interpolator and developing algorithms that outperform existing methods.

Contribution

The paper proposes a novel unlearning framework suitable for overparameterized models, with new algorithms based on gradient orthogonality and theoretical guarantees.

Findings

Outperforms existing unlearning baselines in experiments

Provides exact and approximate unlearning guarantees

Defines unlearning as minimum-complexity interpolation

Abstract

Machine unlearning algorithms aim to remove the influence of specific training samples, ideally recovering the model that would have resulted from training on the remaining data alone. We study unlearning in the overparameterized setting, where many models interpolate the data, and defining the solution as any loss minimizer over the retained set$\unicode{x2013}$as in prior work in the underparameterized setting$\unicode{x2013}$is inadequate, since the original model may already interpolate the retained data and satisfy this condition. In this regime, loss gradients vanish, rendering prior methods based on gradient perturbations ineffective, motivating both new unlearning definitions and algorithms. For this setting, we define the unlearning solution as the minimum-complexity interpolator over the retained data and propose a new algorithmic framework that only requires access to model gradients on the retained set at the original solution. We minimize a regularized objective over perturbations constrained to be orthogonal to these model gradients, a first-order relaxation of the interpolation condition. For different model classes, we provide exact and approximate unlearning guarantees and demonstrate that an implementation of our framework outperforms existing baselines across various unlearning experiments.