Variance-Reduced $(\varepsilon,\delta)-$Unlearning using Forget Set Gradients

TL;DR

This paper introduces VRU, a novel first-order algorithm for machine unlearning that directly uses forget set gradients, providing formal guarantees and improved convergence rates over existing methods.

Contribution

VRU is the first first-order unlearning algorithm to incorporate forget set gradients directly, achieving formal guarantees and better convergence rates.

Findings

VRU provably satisfies $(\

VRU outperforms existing unlearning methods in convergence and effectiveness.

Experiments show VRU's superior performance over certified and empirical baselines.

Abstract

In machine unlearning, $(\varepsilon,\delta)-$unlearning is a popular framework that provides formal guarantees on the effectiveness of the removal of a subset of training data, the forget set, from a trained model. For strongly convex objectives, existing first-order methods achieve $(\varepsilon,\delta)-$unlearning, but they only use the forget set to calibrate injected noise, never as a direct optimization signal. In contrast, efficient empirical heuristics often exploit the forget samples (e.g., via gradient ascent) but come with no formal unlearning guarantees. We bridge this gap by presenting the Variance-Reduced Unlearning (VRU) algorithm. To the best of our knowledge, VRU is the first first-order algorithm that directly includes forget set gradients in its update rule, while provably satisfying ($(\varepsilon,\delta)-$unlearning. We establish the convergence of VRU and show that incorporating the forget set yields strictly improved rates, i.e. a better dependence on the achieved error compared to existing first-order $(\varepsilon,\delta)-$unlearning methods. Moreover, we prove that, in a low-error regime, VRU asymptotically outperforms any first-order method that ignores the forget set.Experiments corroborate our theory, showing consistent gains over both state-of-the-art certified unlearning methods and over empirical baselines that explicitly leverage the forget set.