Machine Unlearning in Low-Dimensional Feature Subspace

TL;DR

This paper introduces LOFT, a low-dimensional feature subspace method for machine unlearning that efficiently removes data influence with minimal data access and reduced privacy risks.

Contribution

LOFT presents a novel low-dimensional feature subspace approach for machine unlearning, optimizing a small projection matrix to efficiently unlearn data without raw data reaccess.

Findings

LOFT achieves significantly lower computational overhead.

LOFT demonstrates superior unlearning performance.

LOFT works across diverse models and datasets.

Abstract

Machine Unlearning (MU) aims at removing the influence of specific data from a pretrained model while preserving performance on the remaining data. In this work, a novel perspective for MU is presented upon low-dimensional feature subspaces, which gives rise to the potentials of separating the remaining and forgetting data herein. This separability motivates our LOFT, a method that proceeds unlearning in a LOw-dimensional FeaTure subspace from the pretrained model skithrough principal projections, which are optimized to maximally capture the information of the remaining data and meanwhile diminish that of the forgetting data. In training, LOFT simply optimizes a small-size projection matrix flexibly plugged into the pretrained model, and only requires one-shot feature fetching from the pretrained backbone instead of repetitively accessing the raw data. Hence, LOFT mitigates two critical issues in mainstream MU methods, i.e., the privacy leakage risk from massive data reload and the inefficiency of updates to the entire pretrained model. Extensive experiments validate the significantly lower computational overhead and superior unlearning performance of LOFT across diverse models, datasets, tasks, and applications. Code is anonymously available at https://anonymous.4open.science/r/4352/.