MoreauPruner: Robust Pruning of Large Language Models against Weight Perturbations

TL;DR

This paper introduces MoreauPruner, a robust pruning method for large language models that maintains accuracy despite weight perturbations, addressing instability issues in existing pruning techniques.

Contribution

The paper proposes a new structural pruning method based on the neural network's Moreau envelope, with provable robustness against weight perturbations in large language models.

Findings

MoreauPruner is robust against weight perturbations.

It achieves competitive accuracy compared to existing pruning methods.

The method is validated on multiple well-known LLMs.

Abstract

Few-shot gradient methods have been extensively utilized in existing model pruning methods, where the model weights are regarded as static values and the effects of potential weight perturbations are not considered. However, the widely used large language models (LLMs) have several billion model parameters, which could increase the fragility of few-shot gradient pruning. In this work, we experimentally show that one-shot gradient pruning algorithms could lead to unstable results under perturbations to model weights. And the minor error of switching between data formats bfloat16 and float16 could result in drastically different outcomes. To address such instabilities, we leverage optimization analysis and propose an LLM structural pruning method, called MoreauPruner, with provable robustness against weight perturbations. In MoreauPruner, the model weight importance is estimated based on the neural network's Moreau envelope, which can be flexibly combined with $\ell_1$-norm regularization techniques to induce the sparsity required in the pruning task. We extensively evaluate the MoreauPruner algorithm on several well-known LLMs, including LLaMA-7B, LLaMA-13B, LLaMA3-8B, and Vicuna-7B. Our numerical results suggest the robustness of MoreauPruner against weight perturbations, and indicate the MoreauPruner's successful accuracy-based scores in comparison to several existing pruning methods. We have released the code in \url{https://github.com/ShiningSord/MoreauPruner}.