DUSK: Do Not Unlearn Shared Knowledge

TL;DR

DUSK introduces a benchmark for evaluating machine unlearning in realistic scenarios with overlapping data, highlighting current methods' limitations in selectively removing content without damaging shared knowledge.

Contribution

We propose DUSK, a new benchmark with evaluation metrics for unlearning methods in realistic overlapping data scenarios, and analyze recent methods' effectiveness and limitations.

Findings

Most unlearning methods remove surface text but not deeper knowledge.

Existing methods often damage shared factual content during unlearning.

DUSK provides a realistic evaluation framework for unlearning techniques.

Abstract

Large language models (LLMs) are increasingly deployed in real-world applications, raising concerns about the unauthorized use of copyrighted or sensitive data. Machine unlearning aims to remove such 'forget' data while preserving utility and information from the 'retain' set. However, existing evaluations typically assume that forget and retain sets are fully disjoint, overlooking realistic scenarios where they share overlapping content. For instance, a news article may need to be unlearned, even though the same event, such as an earthquake in Japan, is also described factually on Wikipedia. Effective unlearning should remove the specific phrasing of the news article while preserving publicly supported facts. In this paper, we introduce DUSK, a benchmark designed to evaluate unlearning methods under realistic data overlap. DUSK constructs document sets that describe the same factual content in different styles, with some shared information appearing across all sets and other content remaining unique to each. When one set is designated for unlearning, an ideal method should remove its unique content while preserving shared facts. We define seven evaluation metrics to assess whether unlearning methods can achieve this selective removal. Our evaluation of nine recent unlearning methods reveals a key limitation: while most can remove surface-level text, they often fail to erase deeper, context-specific knowledge without damaging shared content. We release DUSK as a public benchmark to support the development of more precise and reliable unlearning techniques for real-world applications.