GSS: Gated Subspace Steering for Selective Memorization Mitigation in LLMs

TL;DR

This paper introduces Gated Subspace Steering (GSS), a context-aware method that selectively mitigates memorization in large language models, improving privacy and generalization with less computational cost.

Contribution

We propose GSS, a novel, efficient, and effective method for targeted memorization mitigation in LLMs based on a probe-and-steer framework and optimal subspace steering.

Findings

GSS matches or exceeds state-of-the-art memorization reduction.

GSS requires 100-1000x less compute than existing methods.

Provides new theoretical insights into neural memorization geometry.

Abstract

Large language models (LLMs) can memorize and reproduce training sequences verbatim -- a tendency that undermines both generalization and privacy. Existing mitigation methods apply interventions uniformly, degrading performance on the majority of tokens that generalize normally. We show empirically that memorization is sparse, intermittent, and token-conditioned, suggesting that effective mitigation requires context-aware intervention rather than static parameter modification. To this end, we propose a novel and effective selective memorization mitigation method -- Gated Subspace Steering (GSS), which decomposes intervention into a probe (detecting memorization-relevant activations) and a steer (applying targeted correction only when the probe exceeds a threshold). The optimal probe-steer pair emerges from a principled optimization framework based on optimal subspace steering. Experiments on four benchmarks show GSS matches or exceeds state-of-the-art memorization reduction while requiring $100-1000 \times$ less compute than optimization-based alternatives. Furthermore, we provide new theoretical insights into the geometry of memorization in neural representations.