ELUTQ: Optimizing Quantization Accuracy under LUT-Based Computation for Edge LLMs

TL;DR

ELUTQ introduces a novel hierarchical linear quantization method that improves low-bit weight quantization accuracy and efficiency for edge deployment of large language models, reducing hardware requirements and enabling fast inference.

Contribution

The paper proposes HLQ, a new quantization format that captures weight statistics better and eliminates dequantization overhead, with an optimized pipeline for large-scale model quantization and deployment.

Findings

HLQ significantly improves low-bit quantization accuracy.

ELUTQ enables quantization of LLaMA 3.1-70B with limited hardware.

2-bit LLaMA 3.1-8B achieves 1.5x speedup over AWQ.

Abstract

Weight quantization effectively reduces memory consumption and enable the deployment of Large Language Models on edge devices, yet existing hardware-friendly methods often rely on uniform quantization, which suffers from poor weight-distribution fitting and high dequantization overhead under low-bit settings. In this paper, we propose ELUTQ, an efficient quantization framework featuring a novel quantization format termed Hierarchical Linear Quantization (HLQ). HLQ is designed to better capture the statistical characteristics of weights and eliminate dequantization overhead using Bit-serial LUT-based GEMM operations. HLQ significantly improves model accuracy under low-bit settings and achieves performance comparable to QAT methods without any retraining of the weights. Moreover, an optimized quantization pipeline is integrated into ELUTQ, enabling it to complete the quantization of LLaMA 3.1-70B using only 64 GB of CPU memory and 48 GB of VRAM, reducing the hardware requirements for large-scale model quantization. To enable efficient deployment on edge devices, ELUTQ designs high-performance kernels to support end-to-end inference. Our 2-bit LLaMA3.1-8B achieves 1.5x speedup over AWQ on RTX 3090. Code is available at https://github.com/Nkniexin/ELUTQ.