Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA

TL;DR

This paper introduces a SVD-guided orthogonal subspace projection method for continual machine unlearning, effectively preventing interference between sequential unlearning tasks and maintaining model performance.

Contribution

It proposes a static, SVD-based orthogonal projection approach to improve task isolation in continual unlearning, avoiding parameter collision issues of naive LoRA combinations.

Findings

The method maintains high accuracy after 30 unlearning tasks.

It outperforms static fusion in retaining model performance.

Experiments on CIFAR-100 and MNIST validate stability and effectiveness.

Abstract

Continual machine unlearning aims to remove the influence of data that should no longer be retained, while preserving the usefulness of the model on everything else. This setting becomes especially difficult when deletion requests arrive sequentially, because the model must repeatedly adapt without erasing previously retained knowledge. Low-Rank Adaptation (LoRA) offers an efficient way to implement such updates, but naively combining many sequential LoRA modules leads to parameter collision, causing \textit{strong interference} between tasks. We propose a static alternative based on Singular Value Decomposition (SVD)-guided orthogonal subspace projection. Our method constrains each new LoRA update during training so that it lies in the orthogonal complement of the subspaces used by earlier unlearning tasks. This preserves task isolation without requiring dynamic routing at deployment. Experiments on CIFAR-100 with ResNet-20 and on MNIST show stable behavior across long sequences of unlearning tasks. After thirty sequential unlearning tasks, state-of-the-art static fusion reduces retained accuracy from 60.39\% to 12.70\%, whereas the proposed in-training constrained optimization maintains baseline performance ($\sim$58.1\%) while preserving strong unlearning efficacy.