NOVO: Unlearning-Compliant Vision Transformers

TL;DR

NOVO introduces a vision transformer architecture that enables direct, fine-tuning-free unlearning by simulating forget requests during training, effectively erasing specific information while maintaining overall performance.

Contribution

It proposes a novel unlearning-aware transformer that performs on-the-fly unlearning without fine-tuning, using learnable keys to irreversibly erase information.

Findings

Outperforms existing fine-tuning-free unlearning methods.

Effectively erases information confirmed by membership inference attacks.

Maintains high performance on remaining data after unlearning.

Abstract

Machine unlearning (MUL) refers to the problem of making a pre-trained model selectively forget some training instances or class(es) while retaining performance on the remaining dataset. Existing MUL research involves fine-tuning using a forget and/or retain set, making it expensive and/or impractical, and often causing performance degradation in the unlearned model. We introduce {\pname}, an unlearning-aware vision transformer-based architecture that can directly perform unlearning for future unlearning requests without any fine-tuning over the requested set. The proposed model is trained by simulating unlearning during the training process itself. It involves randomly separating class(es)/sub-class(es) present in each mini-batch into two disjoint sets: a proxy forget-set and a retain-set, and the model is optimized so that it is unable to predict the forget-set. Forgetting is achieved by withdrawing keys, making unlearning on-the-fly and avoiding performance degradation. The model is trained jointly with learnable keys and original weights, ensuring withholding a key irreversibly erases information, validated by membership inference attack scores. Extensive experiments on various datasets, architectures, and resolutions confirm {\pname}'s superiority over both fine-tuning-free and fine-tuning-based methods.