Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method

TL;DR

Squeeze10-LLM introduces a staged mixed-precision quantization framework that compresses large language models by 10 times, maintaining high accuracy with ultra low-bit weights through innovative techniques like PBAR and FIAS.

Contribution

It presents a novel post-training quantization method achieving 1.6 bits per weight, significantly improving low-bit LLM performance with two key innovations, PBAR and FIAS.

Findings

Achieves state-of-the-art sub-2bit quantization performance on LLaMA models.

Improves zero-shot classification accuracy from 43% to 56%.

Quantizes 80% of weights to 1 bit and 20% to 4 bits.

Abstract

Deploying large language models (LLMs) is challenging due to their massive parameters and high computational costs. Ultra low-bit quantization can significantly reduce storage and accelerate inference, but extreme compression (i.e., mean bit-width <= 2) often leads to severe performance degradation. To address this, we propose Squeeze10-LLM, effectively "squeezing" 16-bit LLMs' weights by 10 times. Specifically, Squeeze10-LLM is a staged mixed-precision post-training quantization (PTQ) framework and achieves an average of 1.6 bits per weight by quantizing 80% of the weights to 1 bit and 20% to 4 bits. We introduce Squeeze10LLM with two key innovations: Post-Binarization Activation Robustness (PBAR) and Full Information Activation Supervision (FIAS). PBAR is a refined weight significance metric that accounts for the impact of quantization on activations, improving accuracy in low-bit settings. FIAS is a strategy that preserves full activation information during quantization to mitigate cumulative error propagation across layers. Experiments on LLaMA and LLaMA2 show that Squeeze10-LLM achieves state-of-the-art performance for sub-2bit weight-only quantization, improving average accuracy from 43% to 56% on six zero-shot classification tasks--a significant boost over existing PTQ methods. Our code will be released upon publication.