QuantEase: Optimization-based Quantization for Language Models

TL;DR

QuantEase introduces a layer-wise, optimization-based quantization framework for large language models, achieving state-of-the-art compression with minimal accuracy loss and efficient GPU-based implementation.

Contribution

The paper presents a novel coordinate descent algorithm for layer-wise quantization of LLMs, including an outlier-aware variant, enabling near or sub-3-bit quantization with high performance.

Findings

Achieves up to 15% improvement in perplexity and zero-shot accuracy over GPTQ.

Quantizes models like Falcon-180B in about 3 hours on a single GPU.

Outlier-aware approach reaches near or sub-3-bit quantization with minimal accuracy drop.

Abstract

With the rising popularity of Large Language Models (LLMs), there has been an increasing interest in compression techniques that enable their efficient deployment. This study focuses on the Post-Training Quantization (PTQ) of LLMs. Drawing from recent advances, our work introduces QuantEase, a layer-wise quantization framework where individual layers undergo separate quantization. The problem is framed as a discrete-structured non-convex optimization, prompting the development of algorithms rooted in Coordinate Descent (CD) techniques. These CD-based methods provide high-quality solutions to the complex non-convex layer-wise quantization problems. Notably, our CD-based approach features straightforward updates, relying solely on matrix and vector operations, circumventing the need for matrix inversion or decomposition. We also explore an outlier-aware variant of our approach, allowing for retaining significant weights (outliers) with complete precision. Our proposal attains state-of-the-art performance in terms of perplexity and zero-shot accuracy in empirical evaluations across various LLMs and datasets, with relative improvements up to 15% over methods such as GPTQ. Leveraging careful linear algebra optimizations, QuantEase can quantize models like Falcon-180B on a single NVIDIA A100 GPU in $\sim$3 hours. Particularly noteworthy is our outlier-aware algorithm's capability to achieve near or sub-3-bit quantization of LLMs with an acceptable drop in accuracy, obviating the need for non-uniform quantization or grouping techniques, improving upon methods such as SpQR by up to two times in terms of perplexity.