Inference-Time Machine Unlearning via Gated Activation Redirection

TL;DR

GUARD-IT is a novel inference-time unlearning method that uses input-dependent activation steering to remove memorized data from large language models without altering weights, ensuring privacy and safety.

Contribution

It introduces a gradient-free, input-dependent activation redirection technique that unlearns data at inference time, outperforming gradient-based methods in utility and safety.

Findings

Matches or exceeds 12 gradient-based baselines across model scales.

Preserves utility, suppresses memorization, and avoids collapse.

Remains effective under quantization, unlike parameter-editing methods.

Abstract

Large Language Models memorize vast amounts of training data, raising concerns regarding privacy, copyright infringement, and safety. Machine unlearning seeks to remove the influence of a targeted forget set while preserving model performance, ideally approximating a model retrained from scratch without the forget set. Existing approaches aim to achieve this by updating model parameters via gradient-based methods. However, these updates are computationally expensive, lead to irreversible weight changes, and degrade when the model is quantized for deployment. A recent alternative to changing model weights is activation engineering, where activations are changed during inference to steer model behavior. Despite circumventing weight editing, naive activation steering introduces its own failure modes, as a single global steering vector applies the same intervention to every input, leading to unintended changes in model behavior. We introduce Inference-Time Unlearning via Gated Activation Redirection (GUARD-IT), a training- and gradient-free method that unlearns via input-dependent activation steering at inference time. The resulting intervention is applied as a norm-preserving rotation in the residual stream, leaving model weights untouched. Experiments on TOFU and MUSE show that GUARD-IT matches or exceeds 12 gradient-based baselines across three model scales, while being the only method to simultaneously preserve utility, suppress memorization, and avoid catastrophic collapse across all settings. GUARD-IT further supports continual unlearning without retraining, and remains effective under quantization, a scenario in which parameter-editing methods degrade.