AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models

TL;DR

AlphaPruning introduces a theoretically grounded layerwise pruning method for large language models, leveraging spectral density analysis to optimize sparsity ratios and achieve high pruning levels without significant performance loss.

Contribution

It applies Heavy-Tailed Self-Regularization theory to determine layerwise pruning ratios, improving upon heuristic methods for large language model pruning.

Findings

Prunes LLaMA-7B to 80% sparsity with maintained perplexity.

Uses spectral density shape metrics for pruning ratio allocation.

Demonstrates the effectiveness of the method across existing pruning techniques.

Abstract

Recent work on pruning large language models (LLMs) has shown that one can eliminate a large number of parameters without compromising performance, making pruning a promising strategy to reduce LLM model size. Existing LLM pruning strategies typically assign uniform pruning ratios across layers, limiting overall pruning ability; and recent work on layerwise pruning of LLMs is often based on heuristics that can easily lead to suboptimal performance. In this paper, we leverage Heavy-Tailed Self-Regularization (HT-SR) Theory, in particular the shape of empirical spectral densities (ESDs) of weight matrices, to design improved layerwise pruning ratios for LLMs. Our analysis reveals a wide variability in how well-trained, and thus relatedly how prunable, different layers of an LLM are. Based on this, we propose AlphaPruning, which uses shape metrics to allocate layerwise sparsity ratios in a more theoretically principled manner. AlphaPruning can be used in conjunction with multiple existing LLM pruning methods. Our empirical results show that AlphaPruning prunes LLaMA-7B to 80% sparsity while maintaining reasonable perplexity, marking a first in the literature on LLMs. We have open-sourced our code at https://github.com/haiquanlu/AlphaPruning.