LLM Unlearning Should Be Form-Independent

TL;DR

This paper identifies the form-dependent bias in LLM unlearning methods, introduces a benchmark to evaluate robustness, and proposes a training-free technique called ROCR that enhances unlearning effectiveness by targeting invariant concepts.

Contribution

It formally characterizes form-dependent bias in LLM unlearning, introduces ORT benchmark for evaluation, and proposes ROCR, a novel training-free method to improve unlearning robustness.

Findings

Form-dependent bias is widespread and severe in current unlearning methods.

ROCR significantly outperforms traditional unlearning techniques.

ROCR can modify model perceptions within seconds, producing natural outputs.

Abstract

Large Language Model (LLM) unlearning aims to erase or suppress undesirable knowledge within the model, offering promise for controlling harmful or private information to prevent misuse. However, recent studies highlight its limited efficacy in real-world scenarios, hindering practical adoption. In this study, we identify a pervasive issue underlying many downstream failures: the effectiveness of existing unlearning methods heavily depends on the form of training samples and frequently fails to generalize to alternate expressions of the same knowledge. We formally characterize this problem as Form-Dependent Bias and systematically investigate its specific manifestation patterns across various downstream tasks. To quantify its prevalence and support future research, we introduce ORT, a novel benchmark designed to evaluate the robustness of unlearning methods against variations in knowledge expression. Results reveal that Form-Dependent Bias is both widespread and severe among current techniques. We argue that LLM unlearning should be form-independent to address the endless forms of downstream tasks encountered in real-world security-critical scenarios. Towards this goal, we introduce Rank-one Concept Redirection (ROCR), a novel training-free method, as a promising solution path. ROCR performs unlearning by targeting the invariants in downstream tasks, specifically the activated dangerous concepts. It is capable of modifying model parameters within seconds to redirect the model's perception of a specific unlearning target concept to another harmless concept. Extensive experiments demonstrate that ROCR significantly improves unlearning effectiveness compared to traditional methods while generating highly natural outputs.