Prune Once for All: Sparse Pre-Trained Language Models

TL;DR

This paper introduces a method to create sparse, pre-trained Transformer language models through combined pruning and distillation, enabling efficient transfer learning with minimal accuracy loss and further compression via quantization.

Contribution

It presents a novel approach for training sparse pre-trained Transformer models that maintain transferability and achieve high compression ratios with minimal accuracy loss.

Findings

Achieved up to 40x compression with less than 1% accuracy loss.

Created sparse pre-trained BERT and DistilBERT models with state-of-the-art compression ratios.

Demonstrated effective transfer learning on five NLP tasks.

Abstract

Transformer-based language models are applied to a wide range of applications in natural language processing. However, they are inefficient and difficult to deploy. In recent years, many compression algorithms have been proposed to increase the implementation efficiency of large Transformer-based models on target hardware. In this work we present a new method for training sparse pre-trained Transformer language models by integrating weight pruning and model distillation. These sparse pre-trained models can be used to transfer learning for a wide range of tasks while maintaining their sparsity pattern. We demonstrate our method with three known architectures to create sparse pre-trained BERT-Base, BERT-Large and DistilBERT. We show how the compressed sparse pre-trained models we trained transfer their knowledge to five different downstream natural language tasks with minimal accuracy loss. Moreover, we show how to further compress the sparse models' weights to 8bit precision using quantization-aware training. For example, with our sparse pre-trained BERT-Large fine-tuned on SQuADv1.1 and quantized to 8bit we achieve a compression ratio of $40$X for the encoder with less than $1\%$ accuracy loss. To the best of our knowledge, our results show the best compression-to-accuracy ratio for BERT-Base, BERT-Large, and DistilBERT.