DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration

TL;DR

DenoiseRotator enhances pruning robustness in large language models by redistributing parameter importance through learnable transformations, leading to significantly reduced performance degradation under various sparsity constraints.

Contribution

It introduces a novel importance redistribution approach using orthogonal transformations to improve pruning robustness, compatible with existing pruning methods.

Findings

Reduces perplexity gap by 58% on LLaMA3-70B with SparseGPT at 2:4 sparsity.

Improves zero-shot accuracy across multiple models and pruning techniques.

Seamlessly integrates with existing pruning methods to enhance their performance.

Abstract

Pruning is a widely used technique to compress large language models (LLMs) by removing unimportant weights, but it often suffers from significant performance degradation - especially under semi-structured sparsity constraints. Existing pruning methods primarily focus on estimating the importance of individual weights, which limits their ability to preserve critical capabilities of the model. In this work, we propose a new perspective: rather than merely selecting which weights to prune, we first redistribute parameter importance to make the model inherently more amenable to pruning. By minimizing the information entropy of normalized importance scores, our approach concentrates importance onto a smaller subset of weights, thereby enhancing pruning robustness. We instantiate this idea through DenoiseRotator, which applies learnable orthogonal transformations to the model's weight matrices. Our method can be seamlessly integrated with existing pruning techniques such as Magnitude, SparseGPT, and Wanda. Evaluated on LLaMA3, Qwen2.5, and Mistral models under 50% unstructured and 2:4 semi-structured sparsity, DenoiseRotator consistently improves perplexity and zero-shot accuracy. For instance, on LLaMA3-70B pruned with SparseGPT at 2:4 semi-structured sparsity, DenoiseRotator reduces the perplexity gap to the dense model by 58%, narrowing the degradation from 8.1 to 3.4 points. Codes are available at https://github.com/Axel-gu/DenoiseRotator.