Z-Pruner: Post-Training Pruning of Large Language Models for Efficiency without Retraining

TL;DR

Z-Pruner is a novel post-training pruning technique for large language models that reduces size and improves efficiency without retraining, by effectively identifying redundant parameters using weight and activation patterns.

Contribution

It introduces a model-agnostic, efficient pruning method that outperforms existing approaches by leveraging weight and activation information without requiring retraining.

Findings

Z-Pruner achieves lower perplexity scores.

It maintains high zero-shot accuracy.

Outperforms state-of-the-art pruning methods.

Abstract

Large language models (LLMs) have rapidly advanced in recent years, achieving remarkable performance across a wide range of natural language processing tasks. However, this progress has come at the cost of increasingly large model sizes, which pose significant challenges for deployment, scalability, and energy efficiency. To address these limitations, post-training pruning has emerged as a promising approach for reducing model size and inference latency without the need for retraining. Despite these advantages, many existing pruning methods result in substantial performance degradation or require computationally expensive fine-tuning. In this work, we introduce Z-Pruner, a novel post-training pruning method designed to induce sparsity in pretrained LLMs without any retraining. Unlike conventional approaches, Z-Pruner leverages both weight update magnitudes and activation patterns to identify and eliminate redundant parameters more effectively. Our method is model-agnostic, efficient, and easy to implement. We evaluate Z-Pruner using multiple widely-used LLM architectures, including LLaMA-2, LLaMA-3, and OPT, across a diverse set of standard language benchmarks. Experimental results demonstrate that Z-Pruner surpasses state-of-the-art pruning methods that require intensive weight updates. Specifically, Z-Pruner achieves the lowest perplexity scores and the highest overall average score for zero-shot accuracy. We have made the corresponding codes publicly available at https://github.com/sazzadadib/Z-Pruner.