Contrastive Unlearning: A Contrastive Approach to Machine Unlearning
Hong kyu Lee, Qiuchen Zhang, Carl Yang, Jian Lou, Li Xiong
TL;DR
This paper introduces a contrastive unlearning framework that effectively removes the influence of specific training samples from models by optimizing representation spaces, achieving superior unlearning performance and efficiency.
Contribution
It presents a novel contrastive approach to machine unlearning that directly manipulates embeddings to eliminate sample influence without degrading overall model performance.
Findings
Achieves the best unlearning effects compared to state-of-the-art methods.
Maintains high model performance after unlearning.
Demonstrates efficiency across various datasets and models.
Abstract
Machine unlearning aims to eliminate the influence of a subset of training samples (i.e., unlearning samples) from a trained model. Effectively and efficiently removing the unlearning samples without negatively impacting the overall model performance is still challenging. In this paper, we propose a contrastive unlearning framework, leveraging the concept of representation learning for more effective unlearning. It removes the influence of unlearning samples by contrasting their embeddings against the remaining samples so that they are pushed away from their original classes and pulled toward other classes. By directly optimizing the representation space, it effectively removes the influence of unlearning samples while maintaining the representations learned from the remaining samples. Experiments on a variety of datasets and models on both class unlearning and sample unlearning showed…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
- It seems that the contrastive paradigm is new for the field of unlearning. - The experimental results, from both efficiency and efficacy, seem to show the advantages of the proposed contrastive unlearning (while I'm quite confused about *unlearning acc*, which will be elaborated in Weaknesses and Questions).
I'm not familiar with machine unlearning. I read some papers from recent ML conferences, such as ICML, NeurIPS, and CVPR, and try my best to provide a fair review. Some comments may be naive and I'd like to update my score after reading the response from the authors and reviews from other reviewers. Here are my concerns: - I'm quite confused about the setting of single class unlearning. If we expect a model to forget a class, why don't we just add some rules? For a simple classification task, t
1. The paper introduces contrastive learning to the machine unlearning, effectively removing the influence of unlearning samples without significant loss in model performance. Contrastive unlearning is computationally efficient, requiring fewer iterations to achieve the desired unlearning effect compared to other methods. 2. The experiments are comprehensive, including accuracy gap and membership inference attack for unlearning performance. They convinced me of the SOTA performance of the propo
1. Weak scalability compared to non-contrastive methods. The proposed contrastive learning method constructs the positive pair and negative pair in a batch, hence, it might need a large batch size in the practical application which has a large class number. So I suggest using a MoCo-like, more advanced contrasive learning method to further enhance the scalability of the proposed method. 2. The assumption for test samples is a little bit strong, e.g., " If the embeddings of the unlearning sampl
1. The proposed supervised contrastive learning framework is well-structured and easy to understand. By constructing positive and negative pairs, the encoder effectively deactivates the embedding of unlearning samples. 2. The framework performs well across three benchmarks, showing improved unlearning efficiency and effectiveness over existing methods.
1. The approach of using supervised contrastive learning for unlearning is somewhat simplistic, as it overlooks the potential generalization decay of the model, as noted in Question 1. Merely introducing a framework that achieves improved performance is not sufficiently insightful. ICLR standards are high, and this work could benefit from a deeper exploration of these issues for the community. 2. The presentation quality could be enhanced, especially regarding punctuation usage. For example, the
1. The paper is well-organized. 2. The motivation is clearly described. 3. The appendix is comprehensive.
1. Table 1 is confusing. The results for “remain test” on RN18 seem to suggest that higher is better, so why is 85.79 bolded? 2. A similar issue appears in Table 3. 3. The datasets used are only CIFAR-10 and SVHN. It is recommended to include additional datasets. 4. The authors should release the code to ensure reproducibility.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsImage Enhancement Techniques · Face and Expression Recognition · Machine Learning and ELM