LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models

TL;DR

LLM-Barber introduces a novel one-shot pruning method for large language models that optimizes sparsity masks efficiently without retraining, maintaining high performance and reducing computational costs.

Contribution

The paper presents the first use of weight-gradient product as a pruning metric for LLMs, enabling efficient one-shot pruning with block-aware error optimization.

Findings

Prunes LLaMA and OPT models (7B-13B) in 30 minutes on a single GPU.

Achieves state-of-the-art perplexity and zero-shot performance.

Reduces computational complexity compared to second-order methods.

Abstract

Large language models (LLMs) have seen substantial growth, necessitating efficient model pruning techniques. Existing post-training pruning methods primarily measure weight importance in converged dense models, often overlooking changes in weight significance during the pruning process, leading to performance degradation. To address this issue, we present LLM-Barber (Block-Aware Rebuilder for Sparsity Mask in One-Shot), a novel one-shot pruning framework that rebuilds the sparsity mask of pruned models without any retraining or weight reconstruction. LLM-Barber incorporates block-aware error optimization across Self-Attention and MLP blocks, facilitating global performance optimization. We are the first to employ the product of weights and gradients as a pruning metric in the context of LLM post-training pruning. This enables accurate identification of weight importance in massive models and significantly reduces computational complexity compared to methods using secondorder information. Our experiments show that LLM-Barber efficiently prunes models from LLaMA and OPT families (7B to 13B) on a single A100 GPU in just 30 minutes, achieving state-of-the-art results in both perplexity and zero-shot performance across various language benchmarks. Code is available at https://github.com/YupengSu/LLM-Barber.