Model-Preserving Adaptive Rounding

TL;DR

YAQA is a novel adaptive quantization algorithm that directly minimizes end-to-end error, providing theoretical guarantees and outperforming existing methods like GPTQ and LDLQ in reducing quantization error.

Contribution

Introduces YAQA, the first end-to-end error bound-based adaptive rounding algorithm for quantization with theoretical analysis and superior empirical performance.

Findings

YAQA reduces quantization error by approximately 30% compared to GPTQ/LDLQ.

YAQA achieves lower error than quantization aware training.

Provides the first theoretical end-to-end error bounds for quantization algorithms.

Abstract

The goal of quantization is to produce a compressed model whose output distribution is as close to the original model's as possible. To do this tractably, most quantization algorithms minimize the immediate activation error of each layer as a proxy for the end-to-end error. However, this ignores the effect of future layers, making it a poor proxy. In this work, we introduce Yet Another Quantization Algorithm (YAQA), an adaptive rounding algorithm that directly considers the error at the network's output. YAQA introduces a series of theoretical results that culminate in the first end-to-end error bounds for quantization algorithms. First, we characterize the convergence time of adaptive rounding algorithms via the structure of their Hessian approximations. We then show that the end-to-end error can be bounded by the approximation's cosine similarity to the true Hessian. This admits a natural Kronecker-factored approximation with corresponding near-optimal Hessian sketches. YAQA is provably better than GPTQ/LDLQ and empirically reduces the error by $\approx 30\%$ over these methods. YAQA even achieves a lower error than quantization aware training. This translates to state of the art performance on downstream tasks, all while adding no inference overhead.