SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

TL;DR

This paper introduces SparseGPT, a novel pruning method that enables one-shot pruning of large GPT models to at least 50% sparsity with minimal accuracy loss, without retraining, and efficiently handles models like OPT-175B and BLOOM-176B.

Contribution

SparseGPT is the first pruning method capable of achieving high sparsity in large GPT models in one shot without retraining, significantly reducing inference costs.

Findings

Pruned models maintain accuracy with over 50% sparsity.

SparseGPT efficiently prunes models like OPT-175B and BLOOM-176B in under 4.5 hours.

Achieves 60% sparsity with negligible perplexity increase.

Abstract

We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches. The code is available at: https://github.com/IST-DASLab/sparsegpt.