Understanding the Impact of Post-Training Quantization on Large Language Models

TL;DR

This paper investigates how post-training quantization affects large language models' performance, focusing on hyperparameter sensitivity, inference speed, and content quality, revealing insights into 4-bit quantization techniques and their practical implications.

Contribution

It provides a comprehensive analysis of post-training quantization effects on LLMs, comparing 4-bit methods and examining hyperparameter sensitivities and inference speed impacts.

Findings

nf4 and fp4 are equally effective 4-bit quantization methods.

nf4 shows greater resilience to temperature variations in Llama2 models.

Int8 quantization results in slower inference speeds compared to unquantized models.

Abstract

Large language models (LLMs) are rapidly increasing in size, with the number of parameters becoming a key factor in the success of many commercial models, such as ChatGPT, Claude, and Bard. Even the recently released publicly accessible models for commercial usage, such as Falcon and Llama2, come equipped with billions of parameters. This significant increase in the number of parameters makes deployment and operation very costly. The remarkable progress in the field of quantization for large neural networks in general and LLMs in particular, has made these models more accessible by enabling them to be deployed on consumer-grade GPUs. Quantized models generally demonstrate comparable performance levels to their unquantized base counterparts. Nonetheless, there exists a notable gap in our comprehensive understanding of how these quantized models respond to hyperparameters, such as temperature, max new tokens, and topk, particularly for next word prediction. The present analysis reveals that nf4 and fp4 are equally proficient 4-bit quantization techniques, characterized by similar attributes such as inference speed, memory consumption, and the quality of generated content. the study identifies nf4 as displaying greater resilience to temperature variations in the case of the llama2 series of models at lower temperature, while fp4 and fp4-dq proves to be a more suitable choice for falcon series of models. It is noteworthy that, in general, 4-bit quantized models of varying sizes exhibit higher sensitivity to temperature in the range of 0.5 to 0.8, unlike their unquantized counterparts. Additionally, int8 quantization is associated with significantly slower inference speeds, whereas unquantized bfloat16 models consistently yield the fastest inference speeds across models of all sizes.