Train Once, Forget Precisely: Anchored Optimization for Efficient Post-Hoc Unlearning

TL;DR

This paper introduces FAMR, a theoretically grounded and efficient method for post-hoc unlearning in deep image classifiers, enabling removal of specific data influences without full retraining.

Contribution

FAMR frames unlearning as a constrained optimization problem with theoretical bounds, providing a scalable and certifiable approach for efficient data removal in vision models.

Findings

FAMR effectively removes specific data influences with minimal performance loss.

Theoretical bounds relate FAMR's solution to influence-function approximations.

Empirical results show strong performance retention on CIFAR-10 and ImageNet-100.

Abstract

As machine learning systems increasingly rely on data subject to privacy regulation, selectively unlearning specific information from trained models has become essential. In image classification, this involves removing the influence of particular training samples, semantic classes, or visual styles without full retraining. We introduce \textbf{Forget-Aligned Model Reconstruction (FAMR)}, a theoretically grounded and computationally efficient framework for post-hoc unlearning in deep image classifiers. FAMR frames forgetting as a constrained optimization problem that minimizes a uniform-prediction loss on the forget set while anchoring model parameters to their original values via an $\ell_2$ penalty. A theoretical analysis links FAMR's solution to influence-function-based retraining approximations, with bounds on parameter and output deviation. Empirical results on class forgetting tasks using CIFAR-10 and ImageNet-100 demonstrate FAMR's effectiveness, with strong performance retention and minimal computational overhead. The framework generalizes naturally to concept and style erasure, offering a scalable and certifiable route to efficient post-hoc forgetting in vision models.