ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization

TL;DR

ParetoQ introduces a unified framework to compare low-bit quantization methods for large language models, revealing a critical transition between 2 and 3 bits and achieving state-of-the-art results with fewer parameters.

Contribution

The paper presents ParetoQ, the first comprehensive framework for comparing various low-bit quantizations, and demonstrates improved performance and insights into the quantization learning transition.

Findings

A notable transition occurs between 2 and 3 bits in quantization.

ParetoQ surpasses previous methods in accuracy with fewer parameters.

2-bit and 3-bit quantizations maintain competitive performance.

Abstract

The optimal bit-width for achieving the best trade-off between quantized model size and accuracy has been a subject of ongoing debate. While some advocate for 4-bit quantization, others propose that 1.58-bit offers superior results. However, the lack of a cohesive framework for different bits has left such conclusions relatively tenuous. We present ParetoQ, the first unified framework that facilitates rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings. Our findings reveal a notable learning transition between 2 and 3 bits: For 3-bits and above, the fine-tuned models stay close to their original pre-trained distributions, whereas for learning 2-bit networks or below, the representations change drastically. By optimizing training schemes and refining quantization functions, ParetoQ surpasses all previous methods tailored to specific bit widths. Remarkably, our ParetoQ ternary 600M-parameter model even outperforms the previous SoTA ternary 3B-parameter model in accuracy, using only one-fifth of the parameters. Extensive experimentation shows that ternary, 2-bit, and 3-bit quantization maintains comparable performance in the size-accuracy trade-off and generally exceeds 4-bit and binary quantization. Considering hardware constraints, 2-bit quantization offers promising potential for memory reduction and speedup.