Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions

TL;DR

This paper introduces a novel certified machine unlearning algorithm for nonconvex models that rewinds training to an earlier state before removing data, providing strong privacy guarantees and practical efficiency.

Contribution

It presents the first first-order, black-box unlearning method applicable to general nonconvex functions with theoretical guarantees and improved empirical performance.

Findings

Proves $(psilon, elta)$ certified unlearning guarantees.

Establishes generalization bounds under Polyak-Lojasiewicz condition.

Demonstrates superior empirical performance over existing methods.

Abstract

Machine unlearning algorithms aim to efficiently remove data from a model without retraining it from scratch, in order to remove corrupted or outdated data or respect a user's ``right to be forgotten." Certified machine unlearning is a strong theoretical guarantee based on differential privacy that quantifies the extent to which an algorithm erases data from the model weights. In contrast to existing works in certified unlearning for convex or strongly convex loss functions, or nonconvex objectives with limiting assumptions, we propose the first, first-order, black-box (i.e., can be applied to models pretrained with vanilla gradient descent) algorithm for unlearning on general nonconvex loss functions, which unlearns by ``rewinding" to an earlier step during the learning process before performing gradient descent on the loss function of the retained data points. We prove $(\epsilon, \delta)$ certified unlearning and performance guarantees that establish the privacy-utility-complexity tradeoff of our algorithm, and we prove generalization guarantees for functions that satisfy the Polyak-Lojasiewicz inequality. Finally, we demonstrate the superior performance of our algorithm compared to existing methods, within a new experimental framework that more accurately reflects unlearning user data in practice.