FIT to Forget: Robust Continual Unlearning for Large Language Models

TL;DR

This paper introduces it, a robust continual unlearning framework for large language models that effectively handles sequential deletion requests while maintaining utility and resisting catastrophic forgetting.

Contribution

The paper proposes a novel it framework with three mechanisms and a comprehensive benchmark for evaluating continual unlearning in LLMs, advancing the state-of-the-art.

Findings

it outperforms existing methods in unlearning efficacy and utility preservation.

It maintains strong downstream performance after hundreds of sequential requests.

it shows resilience against relearning and recovery attacks.

Abstract

While large language models (LLMs) exhibit remarkable capabilities, they increasingly face demands to unlearn memorized privacy-sensitive, copyrighted, or harmful content. Existing unlearning methods primarily focus on \emph{single-shot} scenarios, whereas real-world deletion requests arrive \emph{continually}. Na\"ively applying these methods to sequential requests leads to severe utility degradation and catastrophic forgetting. To address this, we propose \fit, a robust continual unlearning framework to process high-volume sequential deletion streams while resisting both catastrophic forgetting and post-unlearning recovery. \fit stabilizes sequential updates through three synergistic mechanisms: redundancy \underline{F}iltering, \underline{I}mportance-aware adaptive algorithm selection, and \underline{T}argeted layer attribution. Furthermore, to facilitate rigorous evaluation, we introduce \textbf{PCH}, a unified benchmark encompassing \textbf{P}ersonal, \textbf{C}opyrighted, and \textbf{H}armful content, alongside two symmetric metrics, Forget Degree (F.D.) and Retain Utility (R.U.), to systematically quantify forgetting-utility trade-offs. Extensive experiments across five LLMs (up to 14B parameters) demonstrate that \fit consistently achieves state-of-the-art unlearning efficacy and utility preservation. Notably, even after hundreds of sequential requests, \fit preserves strong downstream (\eg, GSM8K, MMLU) performance and exhibits superior resilience against relearning and quantization recovery attacks.