LoRA Unlearns More and Retains More (Student Abstract)

TL;DR

This paper introduces PruneLoRA, a novel machine unlearning method that combines pruning and low-rank adaptation to efficiently remove class-specific information while maintaining model performance, outperforming existing approximate unlearning techniques.

Contribution

PruneLoRA presents a new unlearning paradigm that leverages LoRA and pruning to reduce computational costs and improve retention of remaining class performance.

Findings

Outperforms other approximate unlearning methods

Reduces computational cost and memory requirements

Bridges the gap between exact and approximate unlearning

Abstract

Due to increasing privacy regulations and regulatory compliance, Machine Unlearning (MU) has become essential. The goal of unlearning is to remove information related to a specific class from a model. Traditional approaches achieve exact unlearning by retraining the model on the remaining dataset, but incur high computational costs. This has driven the development of more efficient unlearning techniques, including model sparsification techniques, which boost computational efficiency, but degrade the model's performance on the remaining classes. To mitigate these issues, we propose a novel method, PruneLoRA which introduces a new MU paradigm, termed prune first, then adapt, then unlearn. LoRA (Hu et al. 2022) reduces the need for large-scale parameter updates by applying low-rank updates to the model. We leverage LoRA to selectively modify a subset of the pruned model's parameters, thereby reducing the computational cost, memory requirements and improving the model's ability to retain performance on the remaining classes. Experimental Results across various metrics showcase that our method outperforms other approximate MU methods and bridges the gap between exact and approximate unlearning. Our code is available at https://github.com/vlgiitr/LoRA-Unlearn.