GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance

TL;DR

GuidedQuant introduces a novel quantization method for large language models that leverages end loss guidance and preserves weight interactions, significantly improving quantization performance without retraining.

Contribution

The paper presents GuidedQuant, a new quantization approach that incorporates end loss gradient information and maintains cross-weight dependencies, along with a non-uniform scalar quantization algorithm.

Findings

Boosts performance of state-of-the-art quantization methods

Outperforms existing non-uniform quantization algorithms

Provides a reproducible implementation available online

Abstract

Post-training quantization is a key technique for reducing the memory and inference latency of large language models by quantizing weights and activations without requiring retraining. However, existing methods either (1) fail to account for the varying importance of hidden features to the end loss or, when incorporating end loss, (2) neglect the critical interactions between model weights. To address these limitations, we propose GuidedQuant, a novel quantization approach that integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies within output channels. GuidedQuant consistently boosts the performance of state-of-the-art quantization methods across weight-only scalar, weight-only vector, and weight-and-activation quantization. Additionally, we introduce a novel non-uniform scalar quantization algorithm, which is guaranteed to monotonically decrease the quantization objective value, and outperforms existing methods in this category. We release the code at https://github.com/snu-mllab/GuidedQuant.