Descend or Rewind? Stochastic Gradient Descent Unlearning

TL;DR

This paper provides theoretical guarantees for stochastic gradient descent unlearning algorithms, specifically D2D and R2D, across different convexity settings, and compares their empirical performance.

Contribution

It offers the first $( ext{}\varepsilon, ext{} ext{}\delta)$ unlearning guarantees for stochastic D2D and R2D algorithms on various loss functions, with a novel analysis framework.

Findings

D2D provides tighter guarantees for strongly convex functions.

R2D is more suitable for convex and nonconvex functions.

Empirical results highlight the strengths and weaknesses of each algorithm.

Abstract

Machine unlearning algorithms aim to remove the impact of selected training data from a model without the computational expenses of retraining from scratch. Two such algorithms are ``Descent-to-Delete" (D2D) and ``Rewind-to-Delete" (R2D), full-batch gradient descent algorithms that are easy to implement and satisfy provable unlearning guarantees. In particular, the stochastic version of D2D is widely implemented as the ``finetuning" unlearning baseline, despite lacking theoretical backing on nonconvex functions. In this work, we prove $(\varepsilon, \delta)$ certified unlearning guarantees for stochastic R2D and D2D for strongly convex, convex, and nonconvex loss functions, by analyzing unlearning through the lens of disturbed or biased gradient systems, which may be contracting, semi-contracting, or expansive respectively. Our argument relies on optimally coupling the random behavior of the unlearning and retraining trajectories, resulting in a sensitivity bound that holds in expectation that yields $(\varepsilon, \delta)$ unlearning. We determine that D2D can yield tighter guarantees for strongly convex functions, but R2D is more appropriate for convex and nonconvex functions. Finally, we compare the algorithms empirically, demonstrating the strengths and weaknesses of each approach.