High-Fidelity Pruning for Large Language Models

TL;DR

This paper introduces a novel entropy-based importance criterion for pruning large language models, improving their efficiency while maintaining high predictive fidelity without additional teacher models.

Contribution

It proposes a simple, effective entropy-based importance measure for Taylor pruning that preserves model performance without extra computational overhead.

Findings

Outperforms existing pruning methods on LLaMA and Qwen models

Maintains high zero-shot performance after pruning

Reduces computational costs compared to self distillation approaches

Abstract

Large Language Models (LLMs) have demonstrated exceptional performance across a wide range of tasks, yet their significant computational and memory requirements present major challenges for deployment. A common approach uses Taylor expansion on the loss function to estimate neuron importance. However, its reliance on one-hot cross entropy loss, a key limitation is that it narrowly assesses importance based only on the probability assigned to the single predicted next token, thereby ignoring the other potential predictions of the original model. An intuitive solution to address this is to employ self distillation criterion for importance evaluation. However, this approach introduces significant computational overhead by requiring a separate teacher model for supervision. To this end, we propose a simple but effective criterion, information entropy of the model's output distribution, to efficiently evaluate importance scores of neurons with Taylor pruning without requirement of additional teacher. Compared to plain cross entropy criterion, it provides a more holistic criterion for Taylor pruning to prune neurons with the least impact on the prediction of model in a global manner, thereby preserving the fidelity of the model's predictive capabilities. Experimental results on extensive zero-shot benchmarks demonstrate that our method consistently outperforms existing pruning methods across the LLaMA and Qwen series models. The source code and trained weights are availabel at https://github.com/visresearch/HFPrune.