Bayesian Inference Forgetting

TL;DR

This paper introduces a Bayesian inference forgetting framework that enables efficient removal of individual data influence from models, ensuring privacy rights without retraining from scratch.

Contribution

It proposes the first Bayesian inference forgetting algorithms for variational inference and MCMC, with theoretical guarantees and practical validation.

Findings

Algorithms effectively remove data influence from models

Theoretical guarantees ensure generalizability

Experiments confirm feasibility on synthetic and real data

Abstract

The right to be forgotten has been legislated in many countries but the enforcement in machine learning would cause unbearable costs: companies may need to delete whole models learned from massive resources due to single individual requests. Existing works propose to remove the knowledge learned from the requested data via its influence function which is no longer naturally well-defined in Bayesian inference. This paper proposes a {\it Bayesian inference forgetting} (BIF) framework to realize the right to be forgotten in Bayesian inference. In the BIF framework, we develop forgetting algorithms for variational inference and Markov chain Monte Carlo. We show that our algorithms can provably remove the influence of single datums on the learned models. Theoretical analysis demonstrates that our algorithms have guaranteed generalizability. Experiments of Gaussian mixture models on the synthetic dataset and Bayesian neural networks on the real-world data verify the feasibility of our methods. The source code package is available at \url{https://github.com/fshp971/BIF}.