Feature-Selective Representation Misdirection for Machine Unlearning

TL;DR

This paper introduces SRMU, a novel activation-editing framework for machine unlearning that selectively suppresses harmful representations in large language models, ensuring safety and compliance with minimal utility loss.

Contribution

The paper proposes SRMU, a new activation-editing method that improves unlearning effectiveness in highly entangled models, outperforming existing techniques in safety and utility preservation.

Findings

SRMU achieves state-of-the-art unlearning performance.

Effective under 20-30% dataset overlap where baselines fail.

Minimal utility loss during unlearning process.

Abstract

As large language models (LLMs) are increasingly adopted in safety-critical and regulated sectors, the retention of sensitive or prohibited knowledge introduces escalating risks, ranging from privacy leakage to regulatory non-compliance to to potential misuse, and so on. Recent studies suggest that machine unlearning can help ensure deployed models comply with evolving legal, safety, and governance requirements. However, current unlearning techniques assume clean separation between forget and retain datasets, which is challenging in operational settings characterized by highly entangled distributions. In such scenarios, perturbation-based methods often degrade general model utility or fail to ensure safety. To address this, we propose Selective Representation Misdirection for Unlearning (SRMU), a novel principled activation-editing framework that enforces feature-aware and directionally controlled perturbations. Unlike indiscriminate model weights perturbations, SRMU employs a structured misdirection vector with an activation importance map. The goal is to allow SRMU selectively suppresses harmful representations while preserving the utility on benign ones. Experiments are conducted on the widely used WMDP benchmark across low- and high-entanglement configurations. Empirical results reveal that SRMU delivers state-of-the-art unlearning performance with minimal utility losses, and remains effective under 20-30\% overlap where existing baselines collapse. SRMU provides a robust foundation for safety-driven model governance, privacy compliance, and controlled knowledge removal in the emerging LLM-based applications. We release the replication package at https://figshare.com/s/d5931192a8824de26aff.