Attribute-to-Delete: Machine Unlearning via Datamodel Matching

TL;DR

This paper introduces Datamodel Matching, a new machine unlearning method that leverages data attribution to produce models indistinguishable from retrained models, showing strong empirical results especially in non-convex settings.

Contribution

The paper proposes a novel meta-algorithm, Datamodel Matching, which reduces unlearning to data attribution and demonstrates improved performance over existing methods.

Findings

DMM outperforms existing unlearning algorithms in convex settings.

DMM achieves strong unlearning performance in non-convex models.

Future data attribution advances can enhance unlearning techniques.

Abstract

Machine unlearning -- efficiently removing the effect of a small "forget set" of training data on a pre-trained machine learning model -- has recently attracted significant research interest. Despite this interest, however, recent work shows that existing machine unlearning techniques do not hold up to thorough evaluation in non-convex settings. In this work, we introduce a new machine unlearning technique that exhibits strong empirical performance even in such challenging settings. Our starting point is the perspective that the goal of unlearning is to produce a model whose outputs are statistically indistinguishable from those of a model re-trained on all but the forget set. This perspective naturally suggests a reduction from the unlearning problem to that of data attribution, where the goal is to predict the effect of changing the training set on a model's outputs. Thus motivated, we propose the following meta-algorithm, which we call Datamodel Matching (DMM): given a trained model, we (a) use data attribution to predict the output of the model if it were re-trained on all but the forget set points; then (b) fine-tune the pre-trained model to match these predicted outputs. In a simple convex setting, we show how this approach provably outperforms a variety of iterative unlearning algorithms. Empirically, we use a combination of existing evaluations and a new metric based on the KL-divergence to show that even in non-convex settings, DMM achieves strong unlearning performance relative to existing algorithms. An added benefit of DMM is that it is a meta-algorithm, in the sense that future advances in data attribution translate directly into better unlearning algorithms, pointing to a clear direction for future progress in unlearning.