Qronos: Correcting the Past by Shaping the Future... in Post-Training Quantization

TL;DR

Qronos is a novel post-training quantization algorithm that iteratively corrects errors in neural network weights and activations, leading to improved model performance in language generation tasks.

Contribution

Qronos introduces a new iterative, optimization-based approach for post-training quantization that explicitly corrects quantization errors and surpasses existing data-driven methods.

Findings

Qronos outperforms previous state-of-the-art quantization methods on Llama3 models.

Qronos effectively corrects errors in weights, activations, and KV caches.

The algorithm is compatible with various transformation techniques.

Abstract

We introduce Qronos -- a new state-of-the-art post-training quantization algorithm that sequentially rounds and updates neural network weights. Qronos not only explicitly corrects errors due to both weight and activation quantization, but also errors resulting from quantizing previous layers. Our iterative algorithm is based on an interpretable and disciplined optimization framework that subsumes and surpasses existing data-driven approaches. At each step, Qronos alternates between error correction and diffusion via optimal update rules. Importantly, we prove that Qronos admits an efficient implementation that uses the Cholesky decomposition for solving least-squares problems. We also demonstrate that Qronos is compatible with existing transformation techniques such as Hadamard-based incoherence processing and weight-activation scaling equalization, among others. We evaluate Qronos using recent autoregressive language generation models in the Llama3 family; Qronos consistently outperforms previous state-of-the-art adaptive rounding methods when quantizing the weights, activations, and/or KV caches.