Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective

TL;DR

This paper presents a theoretically grounded method for determining layer-wise sparsity in large language models, addressing the reconstruction error explosion issue and achieving significant performance improvements across various models and tasks.

Contribution

We propose a simple, effective sparsity allocation scheme based on a monotonic arithmetic progression, reducing the complexity of layer-wise sparsity determination to a single hyperparameter tuning.

Findings

Reduces perplexity by 52.10 on sparse LLaMA2-7B.

Improves zero-shot accuracy by 10.50%.

Achieves 2.63x and 2.23x speedups on CPU and GPU.

Abstract

In this paper, we address the challenge of determining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective. Specifically, we identify a critical issue of ''$\textbf{reconstruction error explosion}$'' in existing LLMs sparsification methods. This refers to the cumulative effect of reconstruction errors throughout the sparsification process, where errors from earlier layers propagate and amplify in subsequent layers. As a result, the overall reconstruction error increases significantly, leading to a substantial degradation in model performance. Through theoretical analysis, we derive a simple yet effective approach to layer-wise sparsity allocation that mitigates this issue. Our method uses a monotonically increasing arithmetic progression, reducing the process of determining sparsity rates for multiple layers to the determination of a single common difference hyperparameter. Remarkably, this allows for the optimal layer-wise sparsity rates to be identified with just a few trials. Both our theoretical analysis and experimental results demonstrate that this sparsity allocation scheme is near optimal. Extensive experiments show that our method significantly improves the performance of sparse LLMs across various architectures, outperforming existing layer-wise sparsity methods. Furthermore, it enhances the performance of various compression techniques and is applicable to vision and multimodal models. Notably, our method achieves a reduction of 52.10 in perplexity for the 70$\%$ sparse LLaMA2-7B model obtained via Wanda, improves average zero-shot accuracy by 10.50$\%$, and delivers speedups of 2.63$\times$ and 2.23$\times$ on CPU and GPU, respectively.