Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery

TL;DR

This paper introduces an efficient method for machine unlearning that preserves model utility by using implicit gradient surgery, enabling effective removal of sensitive data with minimal utility loss and computational cost.

Contribution

The paper proposes an implicit gradient surgery approach for utility-preserving machine unlearning, offering a computationally efficient solution with theoretical convergence guarantees.

Findings

Outperforms existing baselines in unlearning-utility tradeoff

Achieves efficient unlearning with only one backpropagation

Provides theoretical convergence analysis of the method

Abstract

Machine unlearning (MU) aims to efficiently remove sensitive or harmful memory from a pre-trained model. The key challenge is to balance the potential tradeoff between unlearning efficacy and utility preservation, which involves forgetting undesirable information as defined while maintaining the model's original performance. One potential way to tackle this problem is to use multi-objective optimization to jointly optimize both the unlearning and utility preservation objectives. However, existing multi-objective methods only guarantee finding a Pareto-optimal solution without fine-grained control, which causes under-optimization of the unlearning objective. To this end, we first model MU as a constrained optimization problem, that is, optimizing the unlearning objective under the constraint of a bounded increase for utility loss. We then show that solving this optimization problem is equivalent to unilateral gradient surgery on the unlearning objective. To resolve the additional computational cost brought by gradient surgery, we propose an implicit gradient surgery method, which approximates the solution to the aforementioned constrained optimization problem via only one backpropagation, thereby achieving efficient utility-preserving MU. Theoretically, we provide a tight convergence analysis of the algorithm. Empirically, our extensive experiments show that the proposed algorithm achieves better tradeoff results than existing baselines. Codes are available at https://github.com/anseryuer/EUPMU-Efficient-Utility-Preserving-Machine-Unlearning.