Unlearning via Sparse Representations
Vedant Shah, Frederik Tr\"auble, Ashish Malik, Hugo Larochelle,, Michael Mozer, Sanjeev Arora, Yoshua Bengio, Anirudh Goyal
TL;DR
This paper introduces a nearly compute-free, zero-shot unlearning method using discrete representations, effectively erasing specific data from models with minimal performance impact and significantly reduced computational costs.
Contribution
The paper presents a novel unlearning technique based on a discrete representational bottleneck that is nearly compute-free and outperforms existing methods like SCRUB in efficiency.
Findings
Effective unlearning on CIFAR-10, CIFAR-100, and LACUNA-100 datasets.
Achieves comparable or better performance than SCRUB.
Incurs almost no additional computational cost.
Abstract
Machine \emph{unlearning}, which involves erasing knowledge about a \emph{forget set} from a trained model, can prove to be costly and infeasible by existing techniques. We propose a nearly compute-free zero-shot unlearning technique based on a discrete representational bottleneck. We show that the proposed technique efficiently unlearns the forget set and incurs negligible damage to the model's performance on the rest of the data set. We evaluate the proposed technique on the problem of \textit{class unlearning} using three datasets: CIFAR-10, CIFAR-100, and LACUNA-100. We compare the proposed technique to SCRUB, a state-of-the-art approach which uses knowledge distillation for unlearning. Across all three datasets, the proposed technique performs as well as, if not better than SCRUB while incurring almost no computational cost.
Peer Reviews
Decision·Submitted to ICLR 2024
**Writing**: The paper's logic is clear, and it is easy to follow.
- **Incremental Novelty**: While the application of concepts from DKVB [1] to the context of machine unlearning is interesting, it may appear as an incremental advance rather than a groundbreaking innovation. Nonetheless, the practical integration of these ideas to address real-world challenges is acknowledged as valuable and can sometimes be sufficient to make significant contributions to the field. (Minor points) - **Comparative Analysis**: The paper could benefit from a broader comparison wi
The paper demonstrates strong performance gains, and is computationally inexpensive. The method only requires forward passes since irrelevant key-value pairs are simply pruned unlike previous methods which utilize some form of negative gradients updates or knowledge distillation. The paper is generally an easy read and figures are clear.
In terms of novelty, I think it’s important to point out that this paper is a direct application of Discrete Key-Value Bottleneck (Trauble 2022). 1. DKVB was proposed for class-incremental learning, and class unlearning is quite literally the inverse task. 2. The original work also shows improvements on CIFAR10/100, the same benchmarks in this paper. 3. DKVB improves class-incremental learning since each class can be learned by disjoint key-value pairs, thus the model updates for learning new
1. Zero Shot Unlearning. 2. Low computational cost 3. Model Specific only applicable to models with Discrete Key Value Bottleneck.
1. Even though the whole motivation of this approach is to implement a new approach with better computational efficiency no experiments on computational cost are done. This approach is not measured with current approaches of class unlearning in terms of computational efficiency.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Data Classification · Advanced Neural Network Applications · Image Enhancement Techniques
MethodsSparse Evolutionary Training · Knowledge Distillation