WaterSIC: information-theoretically (near) optimal linear layer quantization

TL;DR

This paper introduces WaterSIC, a novel quantization algorithm for linear layers in neural networks that is nearly optimal according to information theory, significantly improving over existing methods across various models and bit rates.

Contribution

WaterSIC is the first quantization method to approach the IT limit within 0.255 bits, using a novel rate allocation strategy inspired by waterfilling.

Findings

WaterSIC achieves state-of-the-art results on Llama and Qwen models.

It maintains near-IT optimality across all quantization rates from 1 to 4 bits.

The method outperforms popular algorithms like GPTQ in terms of rate discrepancy.

Abstract

This paper considers the problem of converting a given dense linear layer to low precision. The tradeoff between compressed length and output discrepancy is analyzed information theoretically (IT). It is shown that a popular GPTQ algorithm may have an arbitrarily large gap to the IT limit. To alleviate this problem, a novel algorithm, termed ''WaterSIC'', is proposed and is shown to be within a rate gap of 0.255 bits to the IT limit, uniformly over all possible covariance matrices of input activations. The key innovation of WaterSIC's is to allocate different quantization rates to different columns (in-features) of the weight matrix, mimicking the classical IT solution known as ''waterfilling''. Applying WaterSIC to the Llama and Qwen family of LLMs establishes new state-of-the-art performance for all quantization rates from 1 to 4 bits.