UNO: Unlearning via Orthogonalization in Generative models
Pinak Mandal, Georg A. Gottwald
TL;DR
This paper introduces a fast unlearning method for generative models using loss gradient orthogonalization, enabling efficient removal of specific data while preserving model quality across various datasets and model types.
Contribution
The paper presents a novel unlearning algorithm based on loss gradient orthogonalization that significantly speeds up data removal in generative models without sacrificing performance.
Findings
Achieves orders of magnitude faster unlearning than previous methods
Effective across multiple datasets including MNIST, CelebA, and ImageNet-1K
Applicable to diverse generative models like VAEs and diffusion transformers
Abstract
As generative models become increasingly powerful and pervasive, the ability to unlearn specific data, whether due to privacy concerns, legal requirements, or the correction of harmful content, has become increasingly important. Unlike in conventional training, where data are accumulated and knowledge is reinforced, unlearning aims to selectively remove the influence of particular data points without costly retraining from scratch. To be effective and reliable, such algorithms need to achieve (i) forgetting of the undesired data, (ii) preservation of the quality of the generation, (iii) preservation of the influence of the desired training data on the model parameters, and (iv) small number of training steps. We propose fast unlearning algorithms based on loss gradient orthogonalization for unconditional and conditional generative models. We show that our algorithms are able to forget…
Peer Reviews
Decision·ICLR 2026 Conference Withdrawn Submission
- The algorithm is simple and easy to implement. UNO is just standard loss on retain data plus a cosine-style orthogonality penalty, and UNO-S alternates with surgery. Pseudocode and hyper-parameters are provided. - Evaluations across different generative models and datasets demonstrate the general applicability of the framework. - UNO and UNO-S achieve faster convergence in the “time-to-unlearn” metric while maintaining FID quality, indicating practical improvement over the gradient-surgery b
- Limited technical novelty. The conceptual core, enforcing gradient orthogonality between retain and forget losses, is essentially a rephrasing of gradient surgery (Bae et al., 2023; Yu et al., 2020) and projection-residual unlearning (Cao et al., 2022). These works already formalize orthogonal or residual gradient projection to mitigate interference. - Lack of theoretical contribution. The paper asserts that orthogonalization prevents catastrophic forgetting but provides no formal convergence
1. The proposed method introduce a new regularization to further enforce the loss gradient of retain and forget data be orthogonal, to maintain the balance of knowledge removal and utility preservation. 2. The paper is easy to follow and understand, the proposed algorithm is clearly written.
1. The experiments are limited and there is no comparison to existing unlearning work cited in the related works. 2. The experiments only focus on class-level unlearning, no discussion on sample-based unlearning (or half sample in one classes).
- Presents a conceptually clear approach by introducing gradient orthogonalization into generative model unlearning. - Includes algorithm pseudocode and reproducibility details, which enhance transparency. - Provides demonstrations across several datasets and generative architectures.
Severely limited novelty. The orthogonalization-based formulation is conceptually similar to previously established gradient projection and subspace-based unlearning methods. Works such as "Machine Unlearning under Overparameterization" and the survey "Rethinking machine unlearning for large language models" already discuss orthogonal gradient and subspace decoupling techniques. UNO’s proposal appears to be an incremental adaptation rather than a novel framework. Narrow experimental scope and i
1. The paper is relatively simple and easy to follow. 2. The approach extends gradient surgery and multi-task optimization techniques to address known limitations (catastrophic forgetting,.etc) of simpler methods like gradient ascent and GDiff. 3. The evaluation spans datasets of increasing complexity (MNIST, CelebA, ImageNet-1K) and different model architectures (VAEs, diffusion transformers), demonstrating generalizability and scalability.
1. The approach builds heavily on existing gradient surgery methods from multi-task optimization; the paper primarily positions this as an application to unlearning rather than a fundamentally new algorithmic contribution. And it appears to be largely a combination of gradient descent on the retain set and gradient surgery on the forget set, lacking substantial algorithmic innovation beyond straightforward integration of existing techniques. 2. The paper only compares against relatively basic
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Face recognition and analysis · Digital Media Forensic Detection