Towards Aligned Data Forgetting via Twin Machine Unlearning
Zhenxing Niu, Haoxuan Ji, Yuyao Sun, Zheng Lin, Fei Gao, and Yuhang Wang, Haichao Gao
TL;DR
This paper introduces Twin Machine Unlearning (TMU), a novel method that aligns unlearned models with gold models by defining a twin problem, improving data forgetting in machine learning.
Contribution
The paper proposes TMU, a new approach for aligned data forgetting that enhances the similarity between unlearned and gold models through a twin problem formulation.
Findings
Significantly improves model alignment after unlearning
Outperforms existing unlearning methods in experiments
Enhances privacy compliance in machine learning models
Abstract
Modern privacy regulations have spurred the evolution of machine unlearning, a technique enabling a trained model to efficiently forget specific training data. In prior unlearning methods, the concept of "data forgetting" is often interpreted and implemented as achieving zero classification accuracy on such data. Nevertheless, the authentic aim of machine unlearning is to achieve alignment between the unlearned model and the gold model, i.e., encouraging them to have identical classification accuracy. On the other hand, the gold model often exhibits non-zero classification accuracy due to its generalization ability. To achieve aligned data forgetting, we propose a Twin Machine Unlearning (TMU) approach, where a twin unlearning problem is defined corresponding to the original unlearning problem. Consequently, the generalization-label predictor trained on the twin problem can be…
Peer Reviews
Decision·Submitted to ICLR 2025
- The alignment issue in machine unlearning is well-identified and motivated. - The twin model strategy seems feasible in practice.
- The presentation needs to be improved. The authors should provide more detailed explanations and justifications for their choices. For instance, how to set the boundary between hard/easy labels in practice? Why a well-aligned unlearning algorithm should decrease the accuracy on hard samples? - The evaluation is not convincing. The authors should provide more comprehensive experiments and study the key factors that affect the alignment. - The limitations of the proposed method are not well-dis
* The authors devise a new approach to machine unlearning conscious of the computational burden of re-training the target model from scratch. * Using multiple discriminative features to compute the generalisation label for each sample in $D_{f}$ and $D_{test}$. From Table 3, it is clear that no one metric dominates across all the classes. Using multiple features helps bolster the accuracy of the binary classifier used to label the sample as easy or hard to fit. * The method is straightforwa
* TMU heavily relies on $D_{test}$ and $D_f$ being identically distributed. The authors do not elaborate on how the algorithm's efficacy may vary if this assumption does not hold. For example, if the two data sets differ in terms of the distribution of targeted classes, it is unclear how much it will affect the performance of TMU. * The presentation of the Results section could have been better. For some of the tables and figures, e.g. Table 4 and Figure 3 in the main text, it is unclear which
S1. The paper is well-motivated, addressing the problem of enhancing the "alignment" between the performance of the unlearned model and the retrained model. S2. The introduction of the novel concept "Twin Machine Unlearning" is interesting and appealing. S3. Overall, the authors clearly articulate the challenges of their research and the corresponding solutions, although some detailed explanations are missing and require further clarification (see Weaknesses and Questions).
W1. The statement that "In prior unlearning methods, the concept of ‘data forgetting’ is often interpreted and implemented as achieving zero classification accuracy on such data" is somewhat partial and misleading. In fact, only class-level forgetting may require achieving zero classification accuracy on the forget set. For general random data forgetting task, most prior unlearning studies also consider the retrained model (i.e., the gold model in this paper) as the baseline and aim for optimal
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Data Classification · Image Retrieval and Classification Techniques · Digital Imaging for Blood Diseases