Unlearning vs. Obfuscation: Are We Truly Removing Knowledge?

TL;DR

This paper distinguishes true unlearning from obfuscation in large language models, proposing a new evaluation framework and a method called DF-MCQ that effectively removes specific knowledge and enhances model refusal behavior.

Contribution

The paper introduces a formal distinction between unlearning and obfuscation, a probing-based evaluation framework, and a novel unlearning method called DF-MCQ for large language models.

Findings

DF-MCQ achieves over 90% refusal rate in unlearning tasks.

DF-MCQ significantly increases uncertainty on probing questions.

The evaluation framework effectively differentiates genuine unlearning from obfuscation.

Abstract

Unlearning has emerged as a critical capability for large language models (LLMs) to support data privacy, regulatory compliance, and ethical AI deployment. Recent techniques often rely on obfuscation by injecting incorrect or irrelevant information to suppress knowledge. Such methods effectively constitute knowledge addition rather than true removal, often leaving models vulnerable to probing. In this paper, we formally distinguish unlearning from obfuscation and introduce a probing-based evaluation framework to assess whether existing approaches genuinely remove targeted information. Moreover, we propose DF-MCQ, a novel unlearning method that flattens the model predictive distribution over automatically generated multiple-choice questions using KL-divergence, effectively removing knowledge about target individuals and triggering appropriate refusal behaviour. Experimental results demonstrate that DF-MCQ achieves unlearning with over 90% refusal rate and a random choice-level uncertainty that is much higher than obfuscation on probing questions.