Not All Tokens Are Meant to Be Forgotten
Xiangyu Zhou, Yao Qiang, Saleh Zare Zade, Douglas Zytko, Prashant Khanduri, Dongxiao Zhu

TL;DR
This paper introduces the Targeted Information Forgetting (TIF) framework to improve unlearning in large language models by selectively forgetting unwanted information while preserving general knowledge, achieving state-of-the-art results.
Contribution
The paper proposes a novel TIF framework with targeted identification and preference optimization to enhance unlearning effectiveness and utility preservation in LLMs.
Findings
TIF outperforms existing methods on TOFU and MUSE benchmarks.
TIF effectively balances unlearning unwanted info and retaining general knowledge.
State-of-the-art results demonstrate the framework's superiority.
Abstract
Large Language Models (LLMs), pre-trained on massive text corpora, exhibit remarkable human-level language understanding, reasoning, and decision-making abilities. However, they tend to memorize unwanted information, such as private or copyrighted content, raising significant privacy and legal concerns. Unlearning has emerged as a promising solution, but existing methods face a significant challenge of over-forgetting. This issue arises because they indiscriminately suppress the generation of all the tokens in forget samples, leading to a substantial loss of model utility. To overcome this challenge, we introduce the Targeted Information Forgetting (TIF) framework, which consists of (1) a flexible targeted information identifier designed to differentiate between unwanted words (UW) and general words (GW) in the forget samples, and (2) a novel Targeted Preference Optimization approach…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsTopic Modeling · Multimodal Machine Learning Applications · Big Data and Digital Economy
