Large Language Model Pruning

TL;DR

This paper introduces a theoretically grounded neuron pruning method for large language models that enhances explainability and reduces model complexity without sacrificing performance.

Contribution

It proposes a mutual information-based pruning technique with a theoretical foundation, specifically tailored for large-scale language models, and explores differences in pruning criteria between small and large models.

Findings

Pruning criteria are less sensitive in large models.

The proposed method outperforms state-of-the-art pruning techniques.

The approach improves model explainability and efficiency.

Abstract

We surely enjoy the larger the better models for their superior performance in the last couple of years when both the hardware and software support the birth of such extremely huge models. The applied fields include text mining and others. In particular, the success of LLMs on text understanding and text generation draws attention from researchers who have worked on NLP and related areas for years or even decades. On the side, LLMs may suffer from problems like model overfitting, hallucination, and device limitation to name a few. In this work, we suggest a model pruning technique specifically focused on LLMs. The proposed methodology emphasizes the explainability of deep learning models. By having the theoretical foundation, we obtain a trustworthy deep model so that huge models with a massive number of model parameters become not quite necessary. A mutual information-based estimation is adopted to find neurons with redundancy to eliminate. Moreover, an estimator with well-tuned parameters helps to find precise estimation to guide the pruning procedure. At the same time, we also explore the difference between pruning on large-scale models vs. pruning on small-scale models. The choice of pruning criteria is sensitive in small models but not for large-scale models. It is a novel finding through this work. Overall, we demonstrate the superiority of the proposed model to the state-of-the-art models.