Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs

TL;DR

This paper identifies activation spikes in GLU-based LLMs during quantization, which cause performance degradation, and proposes empirical methods to mitigate these spikes, improving quantization accuracy for various modern LLMs.

Contribution

The paper reveals the pattern of activation spikes in GLU variants and introduces two empirical methods, QFeM and QFeP, to effectively mitigate these spikes during quantization.

Findings

Activation spikes occur mainly in early and late layers.

Proposed methods significantly improve quantization performance.

Methods outperform existing techniques like SmoothQuant.

Abstract

Modern large language models (LLMs) have established state-of-the-art performance through architectural improvements, but still require significant computational cost for inference. In an effort to reduce the inference cost, post-training quantization (PTQ) has become a popular approach, quantizing weights and activations to lower precision, such as INT8. In this paper, we reveal the challenges of activation quantization in GLU variants, which are widely used in feed-forward network (FFN) of modern LLMs, such as LLaMA family. The problem is that severe local quantization errors, caused by excessive magnitudes of activation in GLU variants, significantly degrade the performance of the quantized LLM. We denote these activations as activation spikes. Our further observations provide a systematic pattern of activation spikes: 1) The activation spikes occur in the FFN of specific layers, particularly in the early and late layers, 2) The activation spikes are dedicated to a couple of tokens, rather than being shared across a sequence. Based on our observations, we propose two empirical methods, Quantization-free Module (QFeM) and Quantization-free Prefix (QFeP), to isolate the activation spikes during quantization. Our extensive experiments validate the effectiveness of the proposed methods for the activation quantization, especially with coarse-grained scheme, of latest LLMs with GLU variants, including LLaMA-2/3, Mistral, Mixtral, SOLAR, and Gemma. In particular, our methods enhance the current alleviation techniques (e.g., SmoothQuant) that fail to control the activation spikes. Code is available at https://github.com/onnoo/activation-spikes.