Class Unlearning via Depth-Aware Removal of Forget-Specific Directions

TL;DR

This paper introduces DAMP, a novel one-shot method for class unlearning that effectively removes forget-specific information from deep models without retraining, improving selective forgetting and utility preservation.

Contribution

DAMP is a depth-aware, projection-based weight-surgery technique that removes forget directions in a pretrained network without gradient optimization, extending to multi-class scenarios.

Findings

DAMP outperforms prior methods in selective forgetting across multiple datasets.

It better preserves retain-class performance compared to existing approaches.

DAMP reduces residual forget-class structure in deep representations.

Abstract

Machine unlearning aims to remove targeted knowledge from a trained model without the cost of retraining from scratch. In class unlearning, however, reducing accuracy on forget classes does not necessarily imply true forgetting: forgotten information can remain encoded in internal representations, and apparent forgetting may arise from classifier-head suppression rather than representational removal. We show that existing class-unlearning methods often exhibit weak or negative selectivity, preserve forget-class structure in deep representations, or rely heavily on final-layer bias shifts. We then introduce DAMP (Depth-Aware Modulation by Projection), a one-shot, closed-form weight-surgery method that removes forget-specific directions from a pretrained network without gradient-based optimization. At each stage, DAMP computes class prototypes in the input space of the next learnable operator, extracts forget directions as residuals relative to retain-class prototypes, and applies a projection-based update to reduce downstream sensitivity to those directions. To preserve utility, DAMP uses a parameter-free depth-aware scaling rule derived from probe separability, applying smaller edits in early layers and larger edits in deeper layers. The method naturally extends to multi-class forgetting through low-rank subspace removal. Across MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet, and across convolutional and transformer architectures, DAMP more closely resembles the retraining gold standard than some of the prior methods, improving selective forgetting while better preserving retain-class performance and reducing residual forget-class structure in deep layers.