ROKA: Robust Knowledge Unlearning against Adversaries

TL;DR

This paper introduces ROKA, a novel unlearning method that preserves knowledge and defends against indirect unlearning attacks by balancing model information, with theoretical guarantees and effective performance across large models.

Contribution

ROKA is the first to provide a theoretical framework and guarantee for knowledge preservation during unlearning, addressing knowledge contamination and security threats.

Findings

ROKA effectively unlearns targeted data while maintaining overall model accuracy.

ROKA mitigates indirect unlearning attacks by balancing knowledge influence.

Evaluations show ROKA improves security and performance across various large models.

Abstract

The need for machine unlearning is critical for data privacy, yet existing methods often cause Knowledge Contamination by unintentionally damaging related knowledge. Such a degraded model performance after unlearning has been recently leveraged for new inference and backdoor attacks. Most studies design adversarial unlearning requests that require poisoning or duplicating training data. In this study, we introduce a new unlearning-induced attack model, namely indirect unlearning attack, which does not require data manipulation but exploits the consequence of knowledge contamination to perturb the model accuracy on security-critical predictions. To mitigate this attack, we introduce a theoretical framework that models neural networks as Neural Knowledge Systems. Based on this, we propose ROKA, a robust unlearning strategy centered on Neural Healing. Unlike conventional unlearning methods that only destroy information, ROKA constructively rebalances the model by nullifying the influence of forgotten data while strengthening its conceptual neighbors. To the best of our knowledge, our work is the first to provide a theoretical guarantee for knowledge preservation during unlearning. Evaluations on various large models, including vision transformers, multi-modal models, and large language models, show that ROKA effectively unlearns targets while preserving, or even enhancing, the accuracy of retained data, thereby mitigating the indirect unlearning attacks.