IMU: Influence-guided Machine Unlearning

TL;DR

IMU introduces an influence-guided approach to machine unlearning that selectively targets influential data points, improving utility while effectively forgetting specific data in vision and language models.

Contribution

IMU is the first influence-based unlearning method that dynamically reweights updates using influence functions without full Hessian inversion.

Findings

IMU achieves 30% utility improvement over uniform gradient ascent.

IMU effectively forgets targeted data while maintaining model performance.

Extensive experiments validate IMU's efficiency and effectiveness across tasks.

Abstract

Machine Unlearning (MU) aims to selectively erase the influence of specific data points from pretrained models. However, most existing MU methods rely on the retain set to preserve model utility, which is often impractical due to privacy restrictions and storage constraints. While several retain-data-free methods attempt to bypass this using geometric feature shifts or auxiliary statistics, they typically treat forgetting samples uniformly, overlooking their heterogeneous contributions. To address this, we propose \ul{I}nfluence-guided \ul{M}achine \ul{U}nlearning (IMU), a principled method that conducts MU using only the forget set. Departing from uniform Gradient Ascent (GA) or implicit weighting mechanisms, IMU leverages influence functions as an explicit priority signal to allocate unlearning strength. To circumvent the prohibitive cost of full-model Hessian inversion, we introduce a theoretically grounded classifier-level influence approximation. This efficient design allows IMU to dynamically reweight unlearning updates, aggressively targeting samples that most strongly support the forgetting objective while minimizing unnecessary perturbation to retained knowledge. Extensive experiments across vision and language tasks show that IMU achieves highly competitive results. Compared to standard uniform GA, IMU maintains identical unlearning depth while enhancing model utility by an average of 30%, effectively overcoming the inherent utility-forgetting trade-off.