FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs

TL;DR

FineQuant introduces a weight-only quantization technique for large language models that reduces memory usage and speeds up inference with minimal accuracy loss, without requiring additional fine-tuning.

Contribution

The paper presents a novel heuristic approach for fine-grained weight-only quantization applicable to both dense and MoE models, enhancing efficiency without extra training.

Findings

Achieves up to 3.65x throughput improvement on large models

Maintains minimal accuracy degradation with quantization

Supports efficient GPU GEMMs for quantized models

Abstract

Large Language Models (LLMs) have achieved state-of-the-art performance across various language tasks but pose challenges for practical deployment due to their substantial memory requirements. Furthermore, the latest generative models suffer from high inference costs caused by the memory bandwidth bottleneck in the auto-regressive decoding process. To address these issues, we propose an efficient weight-only quantization method that reduces memory consumption and accelerates inference for LLMs. To ensure minimal quality degradation, we introduce a simple and effective heuristic approach that utilizes only the model weights of a pre-trained model. This approach is applicable to both Mixture-of-Experts (MoE) and dense models without requiring additional fine-tuning. To demonstrate the effectiveness of our proposed method, we first analyze the challenges and issues associated with LLM quantization. Subsequently, we present our heuristic approach, which adaptively finds the granularity of quantization, effectively addressing these problems. Furthermore, we implement highly efficient GPU GEMMs that perform on-the-fly matrix multiplication and dequantization, supporting the multiplication of fp16 or bf16 activations with int8 or int4 weights. We evaluate our approach on large-scale open source models such as OPT-175B and internal MoE models, showcasing minimal accuracy loss while achieving up to 3.65 times higher throughput on the same number of GPUs.