What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation

TL;DR

This paper investigates the challenges of quantizing large language models by viewing quantization as perturbations, revealing insights into failure cases and proposing a non-uniform quantization method that maintains performance.

Contribution

It introduces a perturbation-based perspective on quantization, analyzes its impact on LLMs, and develops a simple non-uniform quantization approach that reduces performance loss.

Findings

Perturbation properties correlate with LLM performance.

Uniform quantization failure cases are explained by perturbation analysis.

Non-uniform quantization achieves minimal performance degradation.

Abstract

Quantization has emerged as a promising technique for improving the memory and computational efficiency of large language models (LLMs). Though the trade-off between performance and efficiency is well-known, there is still much to be learned about the relationship between quantization and LLM performance. To shed light on this relationship, we propose a new perspective on quantization, viewing it as perturbations added to the weights and activations of LLMs. We call this approach "the lens of perturbation". Using this lens, we conduct experiments with various artificial perturbations to explore their impact on LLM performance. Our findings reveal several connections between the properties of perturbations and LLM performance, providing insights into the failure cases of uniform quantization and suggesting potential solutions to improve the robustness of LLM quantization. To demonstrate the significance of our findings, we implement a simple non-uniform quantization approach based on our insights. Our experiments show that this approach achieves minimal performance degradation on both 4-bit weight quantization and 8-bit quantization for weights and activations. These results validate the correctness of our approach and highlight its potential to improve the efficiency of LLMs without sacrificing performance.