Machine Unlearning for Streaming Forgetting

TL;DR

This paper introduces a novel streaming unlearning algorithm that efficiently removes knowledge of specific data in models as data removal requests occur over time, formalizing it as a distribution shift problem and providing theoretical guarantees.

Contribution

The paper proposes a new streaming unlearning paradigm with a distribution shift formulation and a novel algorithm that achieves efficient forgetting without access to original data, supported by theoretical analysis.

Findings

The algorithm achieves an $O(\sqrt{T} + V_T)$ regret bound.

Experimental results validate the effectiveness across various models and datasets.

The method operates efficiently in streaming data removal scenarios.

Abstract

Machine unlearning aims to remove knowledge of the specific training data in a well-trained model. Currently, machine unlearning methods typically handle all forgetting data in a single batch, removing the corresponding knowledge all at once upon request. However, in practical scenarios, requests for data removal often arise in a streaming manner rather than in a single batch, leading to reduced efficiency and effectiveness in existing methods. Such challenges of streaming forgetting have not been the focus of much research. In this paper, to address the challenges of performance maintenance, efficiency, and data access brought about by streaming unlearning requests, we introduce a streaming unlearning paradigm, formalizing the unlearning as a distribution shift problem. We then estimate the altered distribution and propose a novel streaming unlearning algorithm to achieve efficient streaming forgetting without requiring access to the original training data. Theoretical analyses confirm an $O(\sqrt{T} + V_T)$ error bound on the streaming unlearning regret, where $V_T$ represents the cumulative total variation in the optimal solution over $T$ learning rounds. This theoretical guarantee is achieved under mild conditions without the strong restriction of convex loss function. Experiments across various models and datasets validate the performance of our proposed method.