Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models

TL;DR

This paper introduces a Shapley Value-based non-uniform pruning method for large language models that assigns different pruning levels to layers based on their contribution, improving efficiency and performance.

Contribution

The paper proposes SV-NUP, a novel non-uniform pruning approach utilizing Shapley Values to quantify layer importance and optimize pruning across large language models.

Findings

Significant perplexity reduction on LLaMA models.

Non-uniform pruning outperforms uniform methods.

Efficient approximation reduces computational overhead.

Abstract

Pruning large language models (LLMs) is a promising solution for reducing model sizes and computational complexity while preserving performance. Traditional layer-wise pruning methods often adopt a uniform sparsity approach across all layers, which leads to suboptimal performance due to the varying significance of individual transformer layers within the model not being accounted for. To this end, we propose the Shapley Value-based Non-Uniform Pruning (SV-NUP) method for LLMs. This approach quantifies the contribution of each transformer layer to the overall model performance, enabling the assignment of tailored pruning budgets to different layers to retain critical parameters. To further improve efficiency, we design the Sliding Window-based Shapley Value approximation method. It substantially reduces computational overhead compared to exact SV calculation methods. Extensive experiments on various LLMs including LLaMA-v1, LLaMA-v2 and OPT demonstrate the effectiveness of the proposed approach. The results reveal that non-uniform pruning significantly enhances the performance of pruned models. Notably, SV-NUP achieves a reduction in perplexity (PPL) of 18.01% and 19.55% on LLaMA-7B and LLaMA-13B, respectively, compared to SparseGPT at 70% sparsity.