LetheViT: Selective Machine Unlearning for Vision Transformers via Attention-Guided Contrastive Learning
Yujia Tong, Tian Zhang, Jingling Yuan, Yuze Wang, Chuang Hu
TL;DR
LetheViT introduces a novel attention-guided contrastive learning approach for selective unlearning in Vision Transformers, enabling effective data removal while maintaining model performance, thus addressing privacy regulation challenges.
Contribution
The paper presents LetheViT, a new contrastive unlearning method specifically designed for ViTs that improves data forgetting efficiency without sacrificing recognition accuracy.
Findings
Achieves state-of-the-art unlearning performance in ViTs.
Effectively balances privacy compliance with model accuracy.
Reveals key ViT characteristics through attention masking experiments.
Abstract
Vision Transformers (ViTs) have revolutionized computer vision tasks with their exceptional performance. However, the introduction of privacy regulations such as GDPR and CCPA has brought new challenges to them. These laws grant users the right to withdraw their data, necessitating not only the deletion of data but also the complete removal of its influence from trained models. Machine unlearning emerges as a critical solution, with exact unlearning being computationally prohibitive and approximate methods offering a more practical approach. This work addresses the particularly challenging scenario of random data forgetting in ViTs, where the model must forget specific samples while retaining others, even within the same class. We first reveal the core characteristics of ViTs through selective masking experiments: when high-attention areas are masked, the model retains its recognition…
Peer Reviews
Decision·Submitted to ICLR 2026
1. The paper try to tackle random data forgetting (sample-level unlearning), which is a challenging and practical unlearning task. 2. The authors validate their method across a wide variety of models and datasets, providing a comprehensive evaluation.
1. The Core Contrastive Unlearning Objective (Eq. 5) is a heuristic solution. The goal of approximate unlearning is to produce an unlearned model $f_{\theta_u}$ that approximates the "gold standard" retrained model $f_{\theta_r}$ (trained from scratch on $D_r$). The proposed contrastive loss does not optimize towards this objective. The "positive" target for the unlearning model's output $Z = f_{\theta_u}(x)$ is $Z_p = f_{\theta_o}(x_m)$, which is the output of the original (pre-forget) model $f
- The proposed approach is intuitive - Experimental results seem to validate that the methodology is a good solution for the proposed problem - The paper is clearly written and easy to follow, for the most part - I appreciate the theoretical analysis on the convergence of the proposed approach
- Experiments are mainly on medium-scale datasets (CIFAR, Tiny-ImageNet). It is unclear how LetheViT performs on large-scale datasets like full ImageNet, which would be crucial for practical deployment. - Performance is evaluated by comparing average metric values (e.g. average test accuracy) between the unlearning approach and the ideal retrain baseline. However, it seems like method that matches the retrain baseline in terms of the point wise measures doesn't fully validate that the model is
* The authors provide analytic motivation (however, it should be further improved, as mentioned in weaknesses) * The proposed methods surpass other baselines up to 2024.
* The provided analytic motivation is not sufficiently persuasive * Both Figure 1 and Table 1 supports the rationales for employing masking strategy during unlearning. However, why such small amount of patches significantly impacts on memorizing and forgetting samples seems not supported. From my perspective, understanding the role of these few patches and their impacts in unlearning are more essential than the methodological details. * Comparison methods are not up-to-date * While curr
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Adversarial Robustness in Machine Learning · Face recognition and analysis