Magnitude Pruning of Large Pretrained Transformer Models with a Mixture Gaussian Prior

TL;DR

This paper introduces a novel magnitude-based pruning method called MGPP that uses a mixture Gaussian prior to effectively prune large pretrained transformer models, maintaining performance across diverse NLP tasks.

Contribution

The paper proposes MGPP, a new pruning algorithm employing a mixture Gaussian prior, which improves model sparsity while preserving expressive power in large NLP transformers.

Findings

MGPP outperforms existing pruning methods at high sparsity levels.

Extensive NLP task evaluations demonstrate MGPP's superior performance.

Theoretical analysis supports the consistency of sparse transformers with MGPP.

Abstract

Large pretrained transformer models have revolutionized modern AI applications with their state-of-the-art performance in natural language processing (NLP). However, their substantial parameter count poses challenges for real-world deployment. To address this, researchers often reduce model size by pruning parameters based on their magnitude or sensitivity. Previous research has demonstrated the limitations of magnitude pruning, especially in the context of transfer learning for modern NLP tasks. In this paper, we introduce a new magnitude-based pruning algorithm called mixture Gaussian prior pruning (MGPP), which employs a mixture Gaussian prior for regularization. MGPP prunes non-expressive weights under the guidance of the mixture Gaussian prior, aiming to retain the model's expressive capability. Extensive evaluations across various NLP tasks, including natural language understanding, question answering, and natural language generation, demonstrate the superiority of MGPP over existing pruning methods, particularly in high sparsity settings. Additionally, we provide a theoretical justification for the consistency of the sparse transformer, shedding light on the effectiveness of the proposed pruning method.