Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases

TL;DR

This paper investigates INT4 quantization for transformer models, demonstrating significant latency improvements with minimal accuracy loss for certain models, and analyzing its limitations and compatibility with other compression techniques.

Contribution

The study develops an optimized INT4 inference pipeline and provides insights into its effectiveness, failure cases, and integration with other model compression methods.

Findings

INT4 quantization causes negligible accuracy loss for encoder models.

INT4 inference pipeline achieves up to 8.5x latency speedup.

Decoder-only models experience significant accuracy degradation with INT4.

Abstract

Improving the deployment efficiency of transformer-based language models has been challenging given their high computation and memory cost. While INT8 quantization has recently been shown to be effective in reducing both the memory cost and latency while preserving model accuracy, it remains unclear whether we can leverage INT4 (which doubles peak hardware throughput) to achieve further latency improvement. In this study, we explore the feasibility of employing INT4 weight and activation (W4A4) quantization for language models. Our findings indicate that W4A4 quantization introduces no to negligible accuracy degradation for encoder-only and encoder-decoder models, but causes a significant accuracy drop for decoder-only models. To materialize the performance gain using W4A4, we develop a highly optimized end-to-end W4A4 encoder inference pipeline supporting different quantization strategies. Our INT4 pipeline is $8.5\times$ faster for latency-oriented scenarios and up to $3\times$ for throughput-oriented scenarios compared to the inference of FP16, and improves the SOTA BERT INT8 performance from FasterTransformer by up to $1.7\times$. We provide insights into the failure cases when applying W4A4 to decoder-only models, and further explore the compatibility of INT4 quantization with other compression methods, like pruning and layer reduction.