Fast Exact Unlearning for In-Context Learning Data for LLMs

TL;DR

This paper introduces a method for efficiently and exactly unlearning specific fine-tuning data from large language models using in-context learning and quantized k-means, significantly reducing unlearning costs.

Contribution

The paper presents a novel approach for exact unlearning in LLMs by leveraging in-context learning and quantized k-means, enabling fast unlearning without retraining.

Findings

Unlearning performance is comparable to fine-tuning methods.

Unlearning operations are significantly faster and cost-effective.

Highlights the need for new unlearning cost metrics.

Abstract

Modern machine learning models are expensive to train, and there is a growing concern about the challenge of retroactively removing specific training data. Achieving exact unlearning in deep learning pipelines--producing models as if certain data had never been included in training--remains an open problem. In this paper, we revisit exact unlearning in deep learning and show that for large language models (LLMs) we can efficiently exactly unlearn "fine-tuning data" (the data used to adapt a pre-trained model). This follows from two observations. First, we can use in-context learning to adapt the LLM to the fine-tuning dataset instead of SGD based algorithms. Second, we show that accurate in-context learning can be done with quantized k-means, which allows for effectively constant time unlearning operations. Our evaluation shows that this unlearning recipe has similar performance to fine-tuning alternatives, but vastly reduces the unlearning costs. Our study also highlights the need for new measures of unlearning cost when adapting the learning algorithm to have faster unlearn operations.