Turning LLM Activations Quantization-Friendly

TL;DR

This paper explores how to improve quantization of LLM activations by analyzing outliers, introducing a new metric for quantization difficulty, and proposing a hybrid scaling and rotation method to reduce quantization error.

Contribution

It introduces a novel metric for quantization difficulty and a hybrid scaling-rotation approach to enhance activation quantization in LLMs.

Findings

Channel-wise scaling reduces quantization error.

Rotation transforms outlier distributions.

Hybrid approach improves quantization accuracy.

Abstract

Quantization effectively reduces the serving costs of Large Language Models (LLMs) by speeding up data movement through compressed parameters and enabling faster operations via integer arithmetic. However, activating integer arithmetic requires quantizing both weights and activations, which poses challenges due to the significant outliers in LLMs that increase quantization error. In this work, we investigate these outliers with an emphasis on their effect on layer-wise quantization error, then examine how smoothing and rotation transform the observed values. Our primary contributions include introducing a new metric to measure and visualize quantization difficulty based on channel magnitudes, as well as proposing a hybrid approach that applies channel-wise scaling before rotation, supported by a mathematical formulation of its benefits.