Provable unlearning in topic modeling and downstream tasks

TL;DR

This paper introduces the first provable unlearning algorithms for topic models, enabling efficient removal of data with guarantees, applicable to downstream tasks like retrieval and classification.

Contribution

It provides a novel, computationally efficient unlearning method for topic models with formal guarantees, including after fine-tuning for downstream tasks.

Findings

Unlearning algorithm has computational overhead independent of dataset size.

Model's deletion capacity is quantified, showing how much data can be unlearned.

Unlearning is easier after fine-tuning, without modifying the base model.

Abstract

Machine unlearning algorithms are increasingly important as legal concerns arise around the provenance of training data, but verifying the success of unlearning is often difficult. Provable guarantees for unlearning are often limited to supervised learning settings. In this paper, we provide the first theoretical guarantees for unlearning in the pre-training and fine-tuning paradigm by studying topic models, simple bag-of-words language models that can be adapted to solve downstream tasks like retrieval and classification. First, we design a provably effective unlearning algorithm for topic models that incurs a computational overhead independent of the size of the original dataset. Our analysis additionally quantifies the deletion capacity of the model -- i.e., the number of examples that can be unlearned without incurring a significant cost in model performance. Finally, we formally extend our analyses to account for adaptation to a given downstream task. In particular, we design an efficient algorithm to perform unlearning after fine-tuning the topic model via a linear head. Notably, we show that it is easier to unlearn pre-training data from models that have been fine-tuned to a particular task, and one can unlearn this data without modifying the base model.