Challenges and Pitfalls of Bayesian Unlearning

TL;DR

This paper explores the challenges of Bayesian unlearning, focusing on approximate methods like Laplace approximation and Variational Inference to update model posteriors after data removal, especially in neural network regression.

Contribution

It analyzes the practical applicability of Bayesian unlearning techniques using Laplace and Variational Inference for neural network models.

Findings

Laplace approximation provides a feasible way to perform Bayesian unlearning.

Variational Inference offers a scalable alternative but has limitations.

Practical scenarios reveal challenges in accurately updating posteriors after data removal.

Abstract

Machine unlearning refers to the task of removing a subset of training data, thereby removing its contributions to a trained model. Approximate unlearning are one class of methods for this task which avoid the need to retrain the model from scratch on the retained data. Bayes' rule can be used to cast approximate unlearning as an inference problem where the objective is to obtain the updated posterior by dividing out the likelihood of deleted data. However this has its own set of challenges as one often doesn't have access to the exact posterior of the model parameters. In this work we examine the use of the Laplace approximation and Variational Inference to obtain the updated posterior. With a neural network trained for a regression task as the guiding example, we draw insights on the applicability of Bayesian unlearning in practical scenarios.