Leak@$k$: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding

TL;DR

This paper reveals that current unlearning methods for large language models often fail to fully forget sensitive information under realistic sampling, highlighting the need for more robust unlearning techniques.

Contribution

The authors introduce the leak@$k$ metric to evaluate unlearning reliability and propose the RULE algorithm to improve forgetting in LLMs.

Findings

Existing unlearning methods show persistent knowledge leakage under probabilistic decoding.

The leak@$k$ metric effectively quantifies the likelihood of sensitive information reappearing.

RULE reduces information leakage significantly in the TOFU benchmark.

Abstract

Unlearning in large language models (LLMs) is critical for regulatory compliance and for building ethical generative AI systems that avoid producing private, toxic, illegal, or copyrighted content. Despite rapid progress, in this work we show that \textit{almost all} existing unlearning methods fail to achieve true forgetting in practice. Specifically, while evaluations of these `unlearned' models under deterministic (greedy) decoding often suggest successful knowledge removal using standard benchmarks (as has been done in the literature), we show that sensitive information reliably resurfaces when models are sampled with standard probabilistic decoding. To rigorously capture this vulnerability, we introduce \texttt{leak@$k$}, a new meta-evaluation metric that quantifies the likelihood of forgotten knowledge reappearing when generating $k$ samples from the model under realistic decoding strategies. Using three widely adopted benchmarks, TOFU, MUSE, and WMDP, we conduct the first large-scale, systematic study of unlearning reliability using our newly defined \texttt{leak@$k$} metric. Our findings demonstrate that knowledge leakage persists across methods and tasks, underscoring that current state-of-the-art unlearning techniques provide only limited forgetting and highlighting the urgent need for more robust approaches to LLM unlearning. We propose an algorithm, termed Robust Unlearning under LEak@$k$ metric (\texttt{RULE}), which serves as an initial step toward addressing this concern. We demonstrate that \texttt{RULE} provides an unlearned model for TOFU benchmark with no information leakage for a large number of generation samples.