SAU: Sparsity-Aware Unlearning for LLMs via Gradient Masking and Importance Redistribution

TL;DR

This paper introduces SAU, a novel sparsity-aware unlearning method for large language models that effectively removes sensitive information while maintaining model utility, especially in sparse models.

Contribution

SAU is the first unlearning approach designed specifically for sparse LLMs, decoupling unlearning from sparsification to improve forgetting performance.

Findings

SAU outperforms existing unlearning methods on sparse LLMs.

SAU effectively balances privacy and utility in sparse models.

Empirical results show significant improvement in model forgetting capabilities.

Abstract

Large Language Models (LLMs) inevitably memorize sensitive information during training, posing significant privacy risks. Machine unlearning has emerged as a promising solution to selectively remove such information without full retraining. However, existing methods are designed for dense models and overlook model sparsification, an essential technique for efficient LLM deployment. We find that unlearning effectiveness degrades substantially on sparse models. Through empirical analysis, we reveal that this degradation occurs because existing unlearning methods require updating all parameters, yet sparsification prunes substantial weights to zero, fundamentally limiting the model's forgetting capacity. To address this challenge, we propose Sparsity-Aware Unlearning (SAU), which decouples unlearning from sparsification objectives through gradient masking that redirects updates to surviving weights, combined with importance-aware redistribution to compensate for pruned parameters. Extensive experiments demonstrate that SAU significantly outperforms existing methods on sparse LLMs, achieving effective forgetting while preserving model utility.