GMP*: Well-Tuned Gradual Magnitude Pruning Can Outperform Most BERT-Pruning Methods
Eldar Kurtic, Dan Alistarh

TL;DR
This paper demonstrates that a well-tuned, simple variant of gradual magnitude pruning, called GMP*, can outperform many complex BERT pruning methods, emphasizing the importance of parameter tuning.
Contribution
Introducing GMP*, a simple and general variant of GMP that, with proper tuning, surpasses complex pruning methods for large language models.
Findings
GMP* matches or outperforms state-of-the-art pruning methods.
Proper parameter tuning significantly improves baseline performance.
GMP* enhances the performance of second-order pruning methods.
Abstract
We revisit the performance of the classic gradual magnitude pruning (GMP) baseline for large language models, focusing on the classic BERT benchmark on various popular tasks. Despite existing evidence in the literature that GMP performs poorly, we show that a simple and general variant, which we call GMP*, can match and sometimes outperform more complex state-of-the-art methods. Our results provide a simple yet strong baseline for future work, highlight the importance of parameter tuning for baselines, and even improve the performance of the state-of-the-art second-order pruning method in this setting.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Speech Recognition and Synthesis · Topic Modeling
MethodsAttention Is All You Need · Pruning · Linear Layer · Refunds@Expedia|||How do I get a full refund from Expedia? · Residual Connection · Dropout · Weight Decay · Adam · Dense Connections · Linear Warmup With Linear Decay
