Rethinking Output Alignment For 1-bit Post-Training Quantization of Large Language Models

TL;DR

This paper introduces a novel post-training quantization method for 1-bit large language models, addressing fundamental issues like error accumulation and anisotropic distortion to improve performance.

Contribution

It proposes a new output-driven PTQ approach that explicitly tackles layer-wise error and representation distortion, outperforming existing methods.

Findings

Our method outperforms existing 1-bit PTQ techniques in experiments.

Addressing error accumulation and anisotropic distortion is crucial for effective 1-bit quantization.

The approach maintains computational efficiency while improving model output fidelity.

Abstract

Large Language Models (LLMs) deliver strong performance across a wide range of NLP tasks, but their massive sizes hinder deployment on resource-constrained devices. To reduce their computational and memory burden, various compression techniques have been proposed, including quantization, pruning, and knowledge distillation. Among these, post-training quantization (PTQ) is widely adopted for its efficiency, as it requires no retraining and only a small dataset for calibration, enabling low-cost deployment. Recent advances for post-training quantization have demonstrated that even near 4-bit methods can maintain most of the original model performance. However, 1-bit quantization remains particularly challenging. A common strategy in 1-bit quantization is to determine binary weights by matching full-precision parameters, following a weight-driven criterion. However, this objective is not directly aligned with the quantized model's objective, which is to preserve the model's output behavior under the impact of quantization. A natural alternative is to adopt output-driven criteria that minimize discrepancies in model outputs using calibration data. Surprisingly, naive output-driven approaches often perform even worse in the 1-bit regime. In this paper, we show that this failure arises from two fundamental issues: error accumulation across layers and, more critically, \emph{anisotropic distortion} of the representation space. Based on these insights, we propose a novel PTQ method for 1-bit LLMs that explicitly addresses these issues while maintaining computational efficiency. Extensive experiments demonstrate that our approach consistently outperforms existing 1-bit PTQ methods.