A Comprehensive Study on Quantization Techniques for Large Language Models

TL;DR

This paper provides a comprehensive analysis of quantization techniques to reduce the size and computational requirements of large language models, facilitating their deployment on resource-limited devices.

Contribution

It offers an in-depth review of quantization methods, their mathematical foundations, implementation details, and performance outcomes specifically for large language models.

Findings

Quantization significantly reduces model size and inference time.

Different quantization techniques vary in accuracy and efficiency trade-offs.

Quantization enables deployment of LLMs on resource-constrained devices.

Abstract

Large Language Models (LLMs) have been extensively researched and used in both academia and industry since the rise in popularity of the Transformer model, which demonstrates excellent performance in AI. However, the computational demands of LLMs are immense, and the energy resources required to run them are often limited. For instance, popular models like GPT-3, with 175 billion parameters and a storage requirement of 350 GB, present significant challenges for deployment on resource-constrained IoT devices and embedded systems. These systems often lack the computational capacity to handle such large models. Quantization, a technique that reduces the precision of model values to a smaller set of discrete values, offers a promising solution by reducing the size of LLMs and accelerating inference. In this research, we provide a comprehensive analysis of quantization techniques within the machine learning field, with a particular focus on their application to LLMs. We begin by exploring the mathematical theory of quantization, followed by a review of common quantization methods and how they are implemented. Furthermore, we examine several prominent quantization methods applied to LLMs, detailing their algorithms and performance outcomes.