Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM

TL;DR

This paper introduces Dual Grained Quantization (DGQ), a novel method for efficient low-bit quantization of large language models that balances performance and inference speed, enabling practical deployment.

Contribution

DGQ combines fine-grained and coarse-grained quantization with a two-phase grid search and percentile clipping, improving efficiency and accuracy of LLM quantization.

Findings

Outperforms prior quantization methods across various LLMs and tasks.

Achieves 1.12x memory reduction and 3.24x speedup with a custom kernel.

Enables practical deployment of low-bit LLMs in real-world scenarios.

Abstract

Large Language Models (LLMs) pose significant hardware challenges related to memory requirements and computational ability. There are two mainstream quantization schemes for LLMs: coarse-grained ($\textit{e.g.,}$ channel-wise) quantization and fine-grained ($\textit{e.g.,}$ group-wise) quantization. Fine-grained quantization has smaller quantization loss, consequently achieving superior performance. However, when applied to weight-activation quantization, it disrupts continuous integer matrix multiplication, leading to inefficient inference. In this paper, we introduce Dual Grained Quantization (DGQ), a novel A8W4 quantization for LLM that maintains superior performance while ensuring fast inference speed. DSQ dequantizes the fine-grained INT4 weight into coarse-grained INT8 representation and preform matrix multiplication using INT8 kernels. Besides, we develop a two-phase grid search algorithm to simplify the determination of fine-grained and coarse-grained quantization scales. We also devise a percentile clipping schema for smoothing the activation outliers without the need for complex optimization techniques. Experimental results demonstrate that DGQ consistently outperforms prior methods across various LLM architectures and a wide range of tasks. Remarkably, by our implemented efficient CUTLASS kernel, we achieve $\textbf{1.12}$ $\times$ memory reduction and $\textbf{3.24}$ $\times$ speed gains comparing A16W4 implementation. These advancements enable efficient deployment of A8W4 LLMs for real-world applications.