Empirical Evaluation of Post-Training Quantization Methods for Language Tasks

TL;DR

This paper empirically evaluates post-training quantization methods for BERT models, demonstrating that low-bit quantization can maintain high performance and facilitate deployment in resource-limited settings.

Contribution

The study provides a comprehensive comparison of three PTQ methods on BERT models, highlighting OCS's superior performance and exploring the limits of quantization bits for effective model compression.

Findings

OCS outperforms other PTQ methods in minimizing quantization error.

Low-bit quantized BERT models can outperform 32-bit baselines on some tasks.

BERT models can be quantized to 3 bits with minimal performance loss.

Abstract

Transformer-based architectures like BERT have achieved great success in a wide range of Natural Language tasks. Despite their decent performance, the models still have numerous parameters and high computational complexity, impeding their deployment in resource-constrained environments. Post-Training Quantization (PTQ), which enables low-bit computations without extra training, could be a promising tool. In this work, we conduct an empirical evaluation of three PTQ methods on BERT-Base and BERT-Large: Linear Quantization (LQ), Analytical Clipping for Integer Quantization (ACIQ), and Outlier Channel Splitting (OCS). OCS theoretically surpasses the others in minimizing the Mean Square quantization Error and avoiding distorting the weights' outliers. That is consistent with the evaluation results of most language tasks of GLUE benchmark and a reading comprehension task, SQuAD. Moreover, low-bit quantized BERT models could outperform the corresponding 32-bit baselines on several small language tasks, which we attribute to the alleviation of over-parameterization. We further explore the limit of quantization bit and show that OCS could quantize BERT-Base and BERT-Large to 3-bits and retain 98% and 96% of the performance on the GLUE benchmark accordingly. Moreover, we conduct quantization on the whole BERT family, i.e., BERT models in different configurations, and comprehensively evaluate their performance on the GLUE benchmark and SQuAD, hoping to provide valuable guidelines for their deployment in various computation environments.