Pruning Large Language Models via Accuracy Predictor

TL;DR

This paper introduces a novel pruning method for large language models that uses an accuracy predictor trained on architecture-accuracy pairs to automatically optimize model size while maintaining performance.

Contribution

It proposes a non-neural accuracy predictor for LLM pruning, simplifying the process and improving efficiency over manual feature-based methods.

Findings

Perplexity on Wikitext2 decreased by 9.48%

Perplexity on PTB decreased by 5.76%

Average MMLU accuracy increased by 6.28%

Abstract

Large language models(LLMs) containing tens of billions of parameters (or even more) have demonstrated impressive capabilities in various NLP tasks. However, substantial model size poses challenges to training, inference, and deployment so that it is necessary to compress the model. At present, most model compression for LLMs requires manual design of pruning features, which has problems such as complex optimization pipeline and difficulty in retaining the capabilities of certain parts of the model.Therefore, we propose a novel pruning approach: firstly, a training set of a certain number of architecture-accuracy pairs is established, and then a non-neural model is trained as an accuracy predictor. Using the accuracy predictor to further optimize the search space and search, the optimal model can be automatically selected. Experiments show that our proposed approach is effective and efficient. Compared with the baseline, the perplexity(PPL) on Wikitext2 and PTB dropped by 9.48% and 5,76% respectively, and the average accuracy of MMLU increased by 6.28%.