FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Model

TL;DR

FALCON introduces a novel, fine-grained unlearning method for large language models that effectively removes sensitive information while preserving utility, using contrastive orthogonal unalignment guided by information theory.

Contribution

The paper presents FALCON, a new representation-guided unlearning approach that improves precision and effectiveness in removing knowledge from large language models.

Findings

FALCON outperforms existing methods in unlearning effectiveness.

FALCON maintains high model utility after unlearning.

FALCON resists knowledge recovery attacks.

Abstract

Large language models have been widely applied, but can inadvertently encode sensitive or harmful information, raising significant safety concerns. Machine unlearning has emerged to alleviate this concern; however, existing training-time unlearning approaches, relying on coarse-grained loss combinations, have limitations in precisely separating knowledge and balancing removal effectiveness with model utility. In contrast, we propose Fine-grained Activation manipuLation by Contrastive Orthogonal uNalignment (FALCON), a novel representation-guided unlearning approach that leverages information-theoretic guidance for efficient parameter selection, employs contrastive mechanisms to enhance representation separation, and projects conflict gradients onto orthogonal subspaces to resolve conflicts between forgetting and retention objectives. Extensive experiments demonstrate that FALCON achieves superior unlearning effectiveness while maintaining model utility, exhibiting robust resistance against knowledge recovery attempts.