D$^2$Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs

TL;DR

D$^2$Quant introduces a novel weight-only post-training quantization method for large language models, significantly improving accuracy at sub-4-bit precision by addressing quantization bottlenecks and activation deviations.

Contribution

It proposes Dual-Scale Quantizer and Deviation-Aware Correction to enhance low-bit PTQ accuracy without extra bit-width or hardware changes.

Findings

Achieves superior accuracy at sub-4-bit precision across multiple LLMs.

Effectively mitigates activation distribution shifts during quantization.

Provides a practical framework with open-source code for efficient LLM deployment.

Abstract

Large language models (LLMs) deliver strong performance, but their high compute and memory costs make deployment difficult in resource-constrained scenarios. Weight-only post-training quantization (PTQ) is appealing, as it reduces memory usage and enables practical speedup without low-bit operators or specialized hardware. However, accuracy often degrades significantly in weight-only PTQ at sub-4-bit precision, and our analysis identifies two main causes: (1) down-projection matrices are a well-known quantization bottleneck, but maintaining their fidelity often requires extra bit-width; (2) weight quantization induces activation deviations, but effective correction strategies remain underexplored. To address these issues, we propose D$^2$Quant, a novel weight-only PTQ framework that improves quantization from both the weight and activation perspectives. On the weight side, we design a Dual-Scale Quantizer (DSQ) tailored to down-projection matrices, with an absorbable scaling factor that significantly improves accuracy without increasing the bit budget. On the activation side, we propose Deviation-Aware Correction (DAC), which incorporates a mean-shift correction within LayerNorm to mitigate quantization-induced activation distribution shifts. Extensive experiments across multiple LLM families and evaluation metrics show that D$^2$Quant delivers superior performance for weight-only PTQ at sub-4-bit precision. The code and models will be available at https://github.com/XIANGLONGYAN/D2Quant.