RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning

TL;DR

RapidUn is a novel influence-driven framework that enables efficient and scalable unlearning in large language models by selectively updating parameters based on influence scores.

Contribution

It introduces a fast influence estimation module and adaptive reweighting scheme for effective LLM unlearning, outperforming existing methods in efficiency and stability.

Findings

RapidUn achieves up to 100x efficiency over full retraining.

It outperforms Fisher, GA, and LoReUn on multiple benchmarks.

Effective in both in-distribution and out-of-distribution forgetting.

Abstract

Removing specific data influence from large language models (LLMs) remains challenging, as retraining is costly and existing approximate unlearning methods are often unstable. The challenge is exacerbated when the forget set is small or imbalanced. We introduce RapidUn, an influence-driven and parameter-efficient unlearning framework. It first estimates per-sample influence through a fast estimation module, then maps these scores into adaptive update weights that guide selective parameter updates -- forgetting harmful behavior while retaining general knowledge. On Mistral-7B and Llama-3-8B across Dolly-15k and Alpaca-57k, RapidUn achieves up to 100 times higher efficiency than full retraining and consistently outperforms Fisher, GA, and LoReUn on both in-distribution and out-of-distribution forgetting. These results establish influence-guided parameter reweighting as a scalable and interpretable paradigm for LLM unlearning.