Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs

TL;DR

This paper introduces SignRound, a novel weight quantization method for large language models that uses signed gradient descent to optimize rounding and clipping, achieving high accuracy with minimal tuning.

Contribution

SignRound combines QAT and PTQ advantages, optimizing weight rounding via signed gradient descent in just 200 steps, reducing tuning costs and inference overhead.

Findings

Achieved 6.91% to 33.22% accuracy improvements at 2 bits.

Demonstrated near-lossless 4-bit quantization in most scenarios.

Effective across models and tasks, with minimal tuning effort.

Abstract

Large Language Models (LLMs) have demonstrated exceptional proficiency in language-related tasks, but their deployment poses significant challenges due to substantial memory and storage requirements. Weight-only quantization has emerged as a promising solution, significantly reducing memory and storage needs without sacrificing too much performance. In this study, we introduce SignRound, a method that leverages signed gradient descent (SignSGD) to optimize rounding values and weight clipping in just 200 steps. SignRound integrates the advantages of Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ), delivering exceptional results across 2 to 4 bits while minimizing tuning costs and avoiding additional inference overhead. For example, SignRound achieved absolute average accuracy improvements ranging from 6.91% to 33.22% at 2bits, as measured by the average zero-shot accuracy across 11 tasks. It also demonstrates strong generalization in recent models, achieving near-lossless 4-bit quantization in most scenarios. The source code is publicly available at https://github.com/intel/auto-round.