Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-Of-Distribution Images

TL;DR

This paper presents SCAR, a novel unlearning method that effectively forgets specific data without using a retain set, maintaining model performance through out-of-distribution knowledge distillation and a self-forget variant.

Contribution

SCAR introduces a class and architecture-agnostic unlearning approach that eliminates the need for a retain set, using Mahalanobis distance and knowledge distillation with OOD images.

Findings

Outperforms retain set-free methods in unlearning effectiveness

Achieves comparable performance to retain set-based methods

Validated on three public datasets

Abstract

In this paper, we introduce Selective-distillation for Class and Architecture-agnostic unleaRning (SCAR), a novel approximate unlearning method. SCAR efficiently eliminates specific information while preserving the model's test accuracy without using a retain set, which is a key component in state-of-the-art approximate unlearning algorithms. Our approach utilizes a modified Mahalanobis distance to guide the unlearning of the feature vectors of the instances to be forgotten, aligning them to the nearest wrong class distribution. Moreover, we propose a distillation-trick mechanism that distills the knowledge of the original model into the unlearning model with out-of-distribution images for retaining the original model's test performance without using any retain set. Importantly, we propose a self-forget version of SCAR that unlearns without having access to the forget set. We experimentally verified the effectiveness of our method, on three public datasets, comparing it with state-of-the-art methods. Our method obtains performance higher than methods that operate without the retain set and comparable w.r.t the best methods that rely on the retain set.