BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap

TL;DR

BLUR is a new benchmark for evaluating large language model unlearning that realistically assesses forget-retain overlap and reveals limitations of current methods, emphasizing the need for more robust unlearning techniques.

Contribution

We introduce BLUR, a comprehensive benchmark that improves evaluation of LLM unlearning by addressing forget-retain overlap and including diverse, realistic scenarios.

Findings

Existing unlearning methods perform poorly on BLUR.

Simple approaches outperform recent complex methods on BLUR.

Evaluation reveals significant performance drops, highlighting robustness issues.

Abstract

Machine unlearning has the potential to improve the safety of large language models (LLMs) by removing sensitive or harmful information post hoc. A key challenge in unlearning involves balancing between forget quality (effectively unlearning undesirable information) and retain quality (maintaining good performance on other, general tasks). Unfortunately, as we show, current LLM unlearning benchmarks contain highly disparate forget and retain sets -- painting a false picture of the effectiveness of LLM unlearning methods. This can be particularly problematic because it opens the door for benign perturbations, such as relearning attacks, to easily reveal supposedly unlearned knowledge once models are deployed. To address this, we present $\texttt{BLUR}$: a benchmark for LLM unlearning that provides more realistic scenarios of forget-retain overlap. $\texttt{BLUR}$ significantly expands on existing unlearning benchmarks by providing extended evaluation tasks, combined forget/retain queries, and relearning datasets of varying degrees of difficulty. Despite the benign nature of the queries considered, we find that the performance of existing methods drops significantly when evaluated on $\texttt{BLUR}$, with simple approaches performing better on average than more recent methods. These results highlight the importance of robust evaluation and suggest several important directions of future study. Our benchmark is publicly available at: https://huggingface.co/datasets/forgelab/BLUR