GMP*: Well-Tuned Gradual Magnitude Pruning Can Outperform Most BERT-Pruning Methods

TL;DR

This paper demonstrates that a well-tuned, simple variant of gradual magnitude pruning, called GMP*, can outperform many complex BERT pruning methods, emphasizing the importance of parameter tuning.

Contribution

Introducing GMP*, a simple and general variant of GMP that, with proper tuning, surpasses complex pruning methods for large language models.

Findings

GMP* matches or outperforms state-of-the-art pruning methods.

Proper parameter tuning significantly improves baseline performance.

GMP* enhances the performance of second-order pruning methods.

Abstract

We revisit the performance of the classic gradual magnitude pruning (GMP) baseline for large language models, focusing on the classic BERT benchmark on various popular tasks. Despite existing evidence in the literature that GMP performs poorly, we show that a simple and general variant, which we call GMP*, can match and sometimes outperform more complex state-of-the-art methods. Our results provide a simple yet strong baseline for future work, highlight the importance of parameter tuning for baselines, and even improve the performance of the state-of-the-art second-order pruning method in this setting.