Do Unlearning Methods Remove Information from Language Model Weights?

TL;DR

This paper introduces an adversarial evaluation method to determine if unlearning techniques truly remove sensitive information from language model weights, revealing that current methods often only obscure access rather than eliminate the information.

Contribution

The paper proposes a novel adversarial evaluation approach to assess whether unlearning methods effectively remove information from model weights, highlighting their limitations.

Findings

Fine-tuning on accessible facts can recover 88% of pre-unlearning accuracy.

Current unlearning methods often only obscure information, not remove it.

Evaluations based on fine-tuning may overestimate unlearning robustness.

Abstract

Large Language Models' knowledge of how to perform cyber-security attacks, create bioweapons, and manipulate humans poses risks of misuse. Previous work has proposed methods to unlearn this knowledge. Historically, it has been unclear whether unlearning techniques are removing information from the model weights or just making it harder to access. To disentangle these two objectives, we propose an adversarial evaluation method to test for the removal of information from model weights: we give an attacker access to some facts that were supposed to be removed, and using those, the attacker tries to recover other facts from the same distribution that cannot be guessed from the accessible facts. We show that using fine-tuning on the accessible facts can recover 88% of the pre-unlearning accuracy when applied to current unlearning methods for information learned during pretraining, revealing the limitations of these methods in removing information from the model weights. Our results also suggest that unlearning evaluations that measure unlearning robustness on information learned during an additional fine-tuning phase may overestimate robustness compared to evaluations that attempt to unlearn information learned during pretraining.