BAQ: Efficient Bit Allocation Quantization for Large Language Models

TL;DR

This paper introduces BAQ, a novel bit allocation quantization method for large language models that optimally assigns bitwidths based on sensitivity, significantly reducing perplexity compared to existing methods.

Contribution

The paper presents a convex optimization-based framework for adaptive bitwidth allocation in quantization, with a closed-form solution and an efficient algorithm called BAQ.

Findings

BAQ outperforms GPTQ with up to 56× lower perplexity.

The method effectively balances loss minimization and computational complexity.

Experimental validation on models from 125M to 30B parameters demonstrates broad applicability.

Abstract

Post-training model quantization is a widely adopted technique for reducing the memory and computational costs of large language models (LLMs). However, most existing methods rely on uniform or heuristic bitwidth assignments, failing to account for the nonuniform sensitivity of weights to quantization noise. In this paper, we propose a novel framework for allocating quantization bitwidths based on sensitivity metrics derived from a Hessian proxy. We make key assumptions, which allow the layer/component-wise loss function to be expressed as an explicit function of the bitwidths. This enables a neat formulation of the bit allocation problem as a convex optimization task, whose closed-form solution adapts precision across weights to minimize the layer-wise quantization loss. Inspecting the solution provides several insights (such as the equal-loss structure), which are then exploited to design the proposed \textbf{BAQ} (Bit Allocation Quantization) algorithm. The proposed algorithm achieves a good trade-off between loss minimization and complexity and allows BAQ to be integrated into standard quantization pipelines with minimal overhead. Experimental results show that BAQ consistently outperforms GPTQ, achieving up to 56$\times$ lower perplexity at the same bitwidth on large language models ranging from 125M to 30B parameters. Leveraging our analytical results derived from solving the optimal bit allocation problem, we also provide a theoretical explanation for the observed gains. All codes of this paper are available at https://github.com/CSU-ModelCompression/BAQ.