BID-LoRA: A Parameter-Efficient Framework for Continual Learning and Unlearning

TL;DR

BID-LoRA is a parameter-efficient framework designed for continual learning and unlearning, enabling precise knowledge deletion, efficient new knowledge integration, and minimal knowledge leakage across cycles.

Contribution

It introduces a unified paradigm for continual learning unlearning and a novel BID-LoRA framework with dedicated adapter pathways and escape unlearning to address knowledge leakage.

Findings

BID-LoRA outperforms existing CLU baselines on CIFAR-100.

It effectively minimizes knowledge leakage across multiple adaptation cycles.

Demonstrates practical applicability in identity management systems.

Abstract

Recent advances in deep learning underscore the need for systems that can not only acquire new knowledge through Continual Learning (CL) but also remove outdated, sensitive, or private information through Machine Unlearning (MU). However, while CL methods are well-developed, MU techniques remain in early stages, creating a critical gap for unified frameworks that depend on both capabilities. We find that naively combining existing CL and MU approaches results in knowledge leakage a gradual degradation of foundational knowledge across repeated adaptation cycles. To address this, we formalize Continual Learning Unlearning (CLU) as a unified paradigm with three key goals: (i) precise deletion of unwanted knowledge, (ii) efficient integration of new knowledge while preserving prior information, and (iii) minimizing knowledge leakage across cycles. We propose Bi-Directional Low-Rank Adaptation (BID-LoRA), a novel framework featuring three dedicated adapter pathways-retain, new, and unlearn applied to attention layers, combined with escape unlearning that pushes forget-class embeddings to positions maximally distant from retained knowledge, updating only 5% of parameters. Experiments on CIFAR-100 show that BID-LoRA outperforms CLU baselines across multiple adaptation cycles. We further evaluate on CASIA-Face100, a curated face recognition subset, demonstrating practical applicability to real-world identity management systems where new users must be enrolled and withdrawn users removed.