Efficient Machine Unlearning via Influence Approximation

TL;DR

This paper introduces Influence Approximation Unlearning (IAU), a novel, efficient approach to machine unlearning that leverages the connection between memorizing and forgetting, significantly reducing computational costs while maintaining model utility.

Contribution

The paper establishes a theoretical link between incremental learning and unlearning, and proposes IAU, an influence approximation method for efficient machine unlearning that outperforms existing techniques.

Findings

IAU achieves better unlearning efficiency and model utility balance.

Extensive experiments show IAU outperforms state-of-the-art methods.

The approach reduces computational overhead in large-scale models.

Abstract

Due to growing privacy concerns, machine unlearning, which aims at enabling machine learning models to ``forget" specific training data, has received increasing attention. Among existing methods, influence-based unlearning has emerged as a prominent approach due to its ability to estimate the impact of individual training samples on model parameters without retraining. However, this approach suffers from prohibitive computational overhead arising from the necessity to compute the Hessian matrix and its inverse across all training samples and parameters, rendering it impractical for large-scale models and scenarios involving frequent data deletion requests. This highlights the difficulty of forgetting. Inspired by cognitive science, which suggests that memorizing is easier than forgetting, this paper establishes a theoretical link between memorizing (incremental learning) and forgetting (unlearning). This connection allows machine unlearning to be addressed from the perspective of incremental learning. Unlike the time-consuming Hessian computations in unlearning (forgetting), incremental learning (memorizing) typically relies on more efficient gradient optimization, which supports the aforementioned cognitive theory. Based on this connection, we introduce the Influence Approximation Unlearning (IAU) algorithm for efficient machine unlearning from the incremental perspective. Extensive empirical evaluations demonstrate that IAU achieves a superior balance among removal guarantee, unlearning efficiency, and comparable model utility, while outperforming state-of-the-art methods across diverse datasets and model architectures. Our code is available at https://github.com/Lolo1222/IAU.