Channel-Wise Mixed-Precision Quantization for Large Language Models

TL;DR

This paper introduces CMPQ, a channel-wise mixed-precision quantization method for large language models that adapts precision levels per channel to reduce memory usage while maintaining performance.

Contribution

The paper proposes a novel mixed-precision quantization approach that allocates precision per channel based on activation distributions, improving adaptability and performance of LLMs on edge devices.

Findings

CMPQ improves quantization performance across different LLM sizes.

CMPQ achieves significant performance gains with modest memory increase.

The method effectively preserves critical information during quantization.

Abstract

Large Language Models (LLMs) have demonstrated remarkable success across a wide range of language tasks, but their deployment on edge devices remains challenging due to the substantial memory requirements imposed by their large parameter sizes. Weight-only quantization presents a promising solution to reduce the memory footprint of LLMs. However, existing approaches primarily focus on integer-bit quantization, limiting their adaptability to fractional-bit quantization tasks and preventing the full utilization of available storage space on devices. In this paper, we introduce Channel-Wise Mixed-Precision Quantization (CMPQ), a novel mixed-precision quantization method that allocates quantization precision in a channel-wise pattern based on activation distributions. By assigning different precision levels to different weight channels, CMPQ can adapt to any bit-width constraint. CMPQ employs a non-uniform quantization strategy and incorporates two outlier extraction techniques that collaboratively preserve the critical information, thereby minimizing the quantization loss. Experiments on different sizes of LLMs demonstrate that CMPQ not only enhances performance in integer-bit quantization tasks but also achieves significant performance gains with a modest increase in memory usage. CMPQ thus represents an adaptive and effective approach to LLM quantization, offering substantial benefits across diverse device capabilities.