FG-OrIU: Towards Better Forgetting via Feature-Gradient Orthogonality for Incremental Unlearning

TL;DR

FG-OrIU introduces a novel framework for incremental unlearning that enforces orthogonality constraints on features and gradients, achieving deep, irreversible forgetting and improved security in pre-trained models.

Contribution

It is the first to unify feature and gradient orthogonality constraints for more effective and irreversible incremental unlearning.

Findings

Outperforms existing methods in deep forgetting effectiveness.

Ensures residual information is unrecoverable, enhancing security.

Maintains balance between forgetting and retention across tasks.

Abstract

Incremental unlearning (IU) is critical for pre-trained models to comply with sequential data deletion requests, yet existing methods primarily suppress parameters or confuse knowledge without explicit constraints on both feature and gradient level, resulting in \textit{superficial forgetting} where residual information remains recoverable. This incomplete forgetting risks security breaches and disrupts retention balance, especially in IU scenarios. We propose FG-OrIU (\textbf{F}eature-\textbf{G}radient \textbf{Or}thogonality for \textbf{I}ncremental \textbf{U}nlearning), the first framework unifying orthogonal constraints on both features and gradients level to achieve deep forgetting, where the forgetting effect is irreversible. FG-OrIU decomposes feature spaces via Singular Value Decomposition (SVD), separating forgetting and remaining class features into distinct subspaces. It then enforces dual constraints: feature orthogonal projection on both forgetting and remaining classes, while gradient orthogonal projection prevents the reintroduction of forgotten knowledge and disruption to remaining classes during updates. Additionally, dynamic subspace adaptation merges newly forgetting subspaces and contracts remaining subspaces, ensuring a stable balance between removal and retention across sequential unlearning tasks. Extensive experiments demonstrate the effectiveness of our method.