From Domains to Instances: Dual-Granularity Data Synthesis for LLM Unlearning

TL;DR

This paper introduces orget, a model-driven data synthesis framework for more effective and efficient evaluation of large language model unlearning at domain and instance levels.

Contribution

It formalizes unlearning granularities and proposes orget, which uses the target model to generate relevant forget sets without external data sources.

Findings

orget improves relevance by ~20 in Harry Potter domain

It increases diversity by ~0.05

It halves data size compared to state-of-the-art methods

Abstract

Although machine unlearning is essential for removing private, harmful, or copyrighted content from LLMs, current benchmarks often fail to faithfully represent the true ``forgetting scope'' learned by the model. We formalize two distinct unlearning granularities, domain-level and instance-level, and propose \BiForget, an automated framework for synthesizing high-quality forget sets. Unlike prior work relying on \emph{external} generators, \BiForget exploits the target model per se to elicit data that matches its internal knowledge distribution through seed-guided and adversarial prompting. Our experiments across diverse benchmarks show that it achieves a superior balance of relevance, diversity, and efficiency. Quantitatively, in the Harry Potter domain, it improves relevance by ${\sim}20$ and diversity by ${\sim}$0.05 while \emph{halving} the total data size compared to SOTAs. Ultimately, it facilitates more robust forgetting and better utility preservation, providing a more rigorous foundation for evaluating LLM unlearning.