Online Forgetting Process for Linear Regression Models

TL;DR

This paper addresses the challenge of online data deletion in linear regression models, proposing algorithms that maintain statistical efficiency under memory constraints and data removal, with theoretical guarantees and empirical validation.

Contribution

It introduces the FIFD-OLS and FIFD-Adaptive Ridge algorithms for online forgetting with theoretical regret bounds and improved performance over fixed regularization methods.

Findings

FIFD-OLS exhibits catastrophic rank swinging due to data deletion.

FIFD-Adaptive Ridge effectively offsets deletion uncertainty.

The proposed methods outperform fixed regularization ridge regression in experiments.

Abstract

Motivated by the EU's "Right To Be Forgotten" regulation, we initiate a study of statistical data deletion problems where users' data are accessible only for a limited period of time. This setting is formulated as an online supervised learning task with \textit{constant memory limit}. We propose a deletion-aware algorithm \texttt{FIFD-OLS} for the low dimensional case, and witness a catastrophic rank swinging phenomenon due to the data deletion operation, which leads to statistical inefficiency. As a remedy, we propose the \texttt{FIFD-Adaptive Ridge} algorithm with a novel online regularization scheme, that effectively offsets the uncertainty from deletion. In theory, we provide the cumulative regret upper bound for both online forgetting algorithms. In the experiment, we showed \texttt{FIFD-Adaptive Ridge} outperforms the ridge regression algorithm with fixed regularization level, and hopefully sheds some light on more complex statistical models.