RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimality

TL;DR

RULE introduces a reinforcement learning framework for targeted unlearning in large language models, effectively removing specific information while maintaining overall utility and improving response naturalness.

Contribution

The paper presents RULE, a novel reinforcement learning-based unlearning method that achieves forget-retain Pareto optimality with minimal data and synthesizes boundary queries for effective model unlearning.

Findings

Outperforms existing methods in forget quality and response naturalness.

Achieves unlearning with only 12% forget set and 8% boundary data.

Enhances model generalization and response naturalness.

Abstract

The widespread deployment of Large Language Models (LLMs) trained on massive, uncurated corpora has raised growing concerns about the inclusion of sensitive, copyrighted, or illegal content. This has led to increasing interest in LLM unlearning: the task of selectively removing specific information from a model without retraining from scratch or degrading overall utility. However, existing methods often rely on large-scale forget and retain datasets, and suffer from unnatural responses, poor generalization, or catastrophic utility loss. In this work, we propose Reinforcement UnLearning (RULE), an efficient framework that formulates unlearning as a refusal boundary optimization problem. RULE is trained with a small portion of the forget set and synthesized boundary queries, using a verifiable reward function that encourages safe refusal on forget--related queries while preserving helpful responses on permissible inputs. We provide both theoretical and empirical evidence demonstrating the effectiveness of RULE in achieving targeted unlearning without compromising model utility. Experimental results show that, with only $12%$ forget set and $8%$ synthesized boundary data, RULE outperforms existing baselines by up to $17.5%$ forget quality and $16.3%$ naturalness response while maintaining general utility, achieving forget--retain Pareto optimality. Remarkably, we further observe that RULE improves the naturalness of model outputs, enhances training efficiency, and exhibits strong generalization ability, generalizing refusal behavior to semantically related but unseen queries.