A unified PAC-Bayesian framework for machine unlearning via information risk minimization

TL;DR

This paper introduces a unified PAC-Bayesian framework for machine unlearning, interpreting existing methods as information risk minimization problems that optimize test loss bounds without retraining.

Contribution

It unifies recent unlearning approaches under a single PAC-Bayesian framework, providing a theoretical basis for their design principles.

Findings

Reinterprets variational unlearning and forgetting Lagrangian as PAC-Bayesian bounds.

Provides a theoretical foundation connecting unlearning methods to information risk minimization.

Offers a new perspective for designing efficient unlearning algorithms.

Abstract

Machine unlearning refers to mechanisms that can remove the influence of a subset of training data upon request from a trained model without incurring the cost of re-training from scratch. This paper develops a unified PAC-Bayesian framework for machine unlearning that recovers the two recent design principles - variational unlearning (Nguyen et.al., 2020) and forgetting Lagrangian (Golatkar et.al., 2020) - as information risk minimization problems (Zhang,2006). Accordingly, both criteria can be interpreted as PAC-Bayesian upper bounds on the test loss of the unlearned model that take the form of free energy metrics.