Compression of Generative Pre-trained Language Models via Quantization

TL;DR

This paper introduces a novel quantization-based compression method for generative pre-trained language models, addressing previous challenges and achieving significant size reduction while maintaining performance.

Contribution

It proposes token-level contrastive distillation and module-wise dynamic scaling to effectively compress generative PLMs, outperforming existing methods.

Findings

Achieves 14.4x compression on GPT-2

Achieves 13.4x compression on BART

Outperforms state-of-the-art compression methods

Abstract

The increasing size of generative Pre-trained Language Models (PLMs) has greatly increased the demand for model compression. Despite various methods to compress BERT or its variants, there are few attempts to compress generative PLMs, and the underlying difficulty remains unclear. In this paper, we compress generative PLMs by quantization. We find that previous quantization methods fail on generative tasks due to the \textit{homogeneous word embeddings} caused by reduced capacity, and \textit{varied distribution of weights}. Correspondingly, we propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different modules. Empirical results on various tasks show that our proposed method outperforms the state-of-the-art compression methods on generative PLMs by a clear margin. With comparable performance with the full-precision models, we achieve 14.4x and 13.4x compression rates on GPT-2 and BART, respectively.