Representation-Aware Unlearning via Activation Signatures: From Suppression to Knowledge-Signature Erasure

TL;DR

This paper introduces KIF, a representation-aware framework for true knowledge erasure in large language models, effectively distinguishing genuine unlearning from surface-level suppression while maintaining model utility.

Contribution

The paper presents a novel architecture that targets internal activation signatures for durable unlearning, overcoming the limitations of prior surface-level suppression methods.

Findings

KIF achieves near-oracle erasure with minimal utility loss.

Standard models show scale-independent true erasure (<3% utility drift).

Reasoning-prior models exhibit architectural divergence affecting unlearning.

Abstract

Selective knowledge erasure from LLMs is critical for GDPR compliance and model safety, yet current unlearning methods conflate behavioral suppression with true knowledge removal, allowing latent capabilities to persist beneath surface-level refusals. In this work, we address this challenge by introducing Knowledge Immunization Framework (KIF), a representation-aware architecture that distinguishes genuine erasure from obfuscation by targeting internal activation signatures rather than surface outputs. Our approach combines dynamic suppression of subject-specific representations with parameter-efficient adaptation, enabling durable unlearning without full model retraining. KIF achieves near-oracle erasure (FQ approx 0.99 vs. 1.00) while preserving utility at oracle levels (MU = 0.62), effectively breaking the stability-erasure tradeoff that has constrained all prior work. We evaluate both standard foundation models (Llama and Mistral) and reasoning-prior models (Qwen and DeepSeek) across 3B to 14B parameters. Our observation shows that standard models exhibit scale-independent true erasure (<3% utility drift), while reasoning-prior models reveal fundamental architectural divergence. Our comprehensive dual-metric evaluation protocol, combining surface-level leakage with latent trace persistence, operationalizes the obfuscation - erasure distinction and enables the first systematic diagnosis of mechanism-level forgetting behavior across model families and scales.