UniErase: Towards Balanced and Precise Unlearning in Language Models

TL;DR

UniErase introduces a novel, precise, and balanced unlearning framework for large language models, effectively removing outdated knowledge while retaining overall model ability, outperforming existing methods across various benchmarks.

Contribution

The paper proposes UniErase, a new editing-based unlearning paradigm using Unlearning Tokens and Edits to achieve precise, balanced unlearning and ability retention in LLMs.

Findings

Outperforms 8 baselines on TOFU benchmark.

Modifies only 3.66% of parameters for effective unlearning.

Achieves 4.01× better model ability retention than previous methods.

Abstract

Large language models (LLMs) require iterative updates to address the outdated information problem, where LLM unlearning offers an approach for selective removal. However, mainstream unlearning methods primarily rely on fine-tuning techniques, which often lack precision in targeted unlearning and struggle to balance unlearning efficacy with general ability under massive and sequential settings. To bridge this gap, in this work, we introduce UniErase, a novel unlearning framework that demonstrates precision and balanced performances between knowledge unlearning and ability retaining. We first propose the Unlearning Token, which is optimized to steer LLMs toward a forgetting space. To achieve concrete unlearning behaviors, we further introduce the lightweight Unlearning Edit to efficiently associate the unlearning targets with this meta-token. Serving as a new unlearning paradigm via editing, UniErase achieves outstanding performances across batch, sequential, and precise unlearning tasks under fictitious and real-world knowledge scenarios. On the TOFU benchmark, compared with 8 baselines, UniErase, modifying only $\sim$ \textbf{3.66%} of the LLM parameters, outperforms the previous best-forgetting baseline by \textbf{$\sim$ 4.01$\times$} for \textbf{model ability} with even higher unlearning efficacy. Similarly, UniErase, with better ability retention, also surpasses the previous best-retaining method by \textbf{35.96%} for \textbf{unlearning efficacy}, showing balanced and dual top-tier performances in the current unlearning community.