TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models

TL;DR

TextPruner is an open-source toolkit that enables efficient, post-training pruning of pre-trained language models, reducing their size and computational requirements without the need for re-training.

Contribution

It introduces structured pruning methods, including vocabulary and transformer pruning, along with a self-supervised approach applicable without labeled data.

Findings

Significant reduction in model size achieved

Effective pruning without re-training

Applicable across various NLP tasks

Abstract

Pre-trained language models have been prevailed in natural language processing and become the backbones of many NLP tasks, but the demands for computational resources have limited their applications. In this paper, we introduce TextPruner, an open-source model pruning toolkit designed for pre-trained language models, targeting fast and easy model compression. TextPruner offers structured post-training pruning methods, including vocabulary pruning and transformer pruning, and can be applied to various models and tasks. We also propose a self-supervised pruning method that can be applied without the labeled data. Our experiments with several NLP tasks demonstrate the ability of TextPruner to reduce the model size without re-training the model.