Unlearning through Knowledge Overwriting: Reversible Federated Unlearning via Selective Sparse Adapter

TL;DR

This paper introduces FUSED, a federated unlearning method that uses sparse adapters for selective, reversible knowledge removal, reducing costs while maintaining effectiveness comparable to retraining.

Contribution

FUSED is a novel federated unlearning approach that identifies sensitive layers, employs sparse adapters for targeted unlearning, and enables reversibility and cost reduction.

Findings

FUSED achieves unlearning effectiveness comparable to retraining.

It significantly reduces unlearning costs.

FUSED enables reversible unlearning through independent adapters.

Abstract

Federated Learning is a promising paradigm for privacy-preserving collaborative model training. In practice, it is essential not only to continuously train the model to acquire new knowledge but also to guarantee old knowledge the right to be forgotten (i.e., federated unlearning), especially for privacy-sensitive information or harmful knowledge. However, current federated unlearning methods face several challenges, including indiscriminate unlearning of cross-client knowledge, irreversibility of unlearning, and significant unlearning costs. To this end, we propose a method named FUSED, which first identifies critical layers by analyzing each layer's sensitivity to knowledge and constructs sparse unlearning adapters for sensitive ones. Then, the adapters are trained without altering the original parameters, overwriting the unlearning knowledge with the remaining knowledge. This knowledge overwriting process enables FUSED to mitigate the effects of indiscriminate unlearning. Moreover, the introduction of independent adapters makes unlearning reversible and significantly reduces the unlearning costs. Finally, extensive experiments on three datasets across various unlearning scenarios demonstrate that FUSED's effectiveness is comparable to Retraining, surpassing all other baselines while greatly reducing unlearning costs.