Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs

TL;DR

This paper investigates how calibration sets and outliers affect the quantization of modern LLMs, revealing that newer models are more robust and suggesting a shift in PTQ strategies towards optimizing inference speed.

Contribution

It demonstrates that the impact of calibration sets and outliers varies across models, with newer LLMs showing robustness, indicating a need to revise existing quantization approaches.

Findings

Older OPT model is highly sensitive to outliers and calibration set variations.

Newer models like Llama-2, Llama-3, Command-R, and Mistral are more robust to outliers.

Mistral 7B shows near-immunity to outliers and stable activations.

Abstract

Post-Training Quantization (PTQ) enhances the efficiency of Large Language Models (LLMs) by enabling faster operation and compatibility with more accessible hardware through reduced memory usage, at the cost of small performance drops. We explore the role of calibration sets in PTQ, specifically their effect on hidden activations in various notable open-source LLMs. Calibration sets are crucial for evaluating activation magnitudes and identifying outliers, which can distort the quantization range and negatively impact performance. Our analysis reveals a marked contrast in quantization effectiveness across models. The older OPT model, upon which much of the quantization literature is based, shows significant performance deterioration and high susceptibility to outliers with varying calibration sets. In contrast, newer models like Llama-2 7B, Llama-3 8B, Command-R 35B, and Mistral 7B demonstrate strong robustness, with Mistral 7B showing near-immunity to outliers and stable activations. These findings suggest a shift in PTQ strategies might be needed. As advancements in pre-training methods reduce the relevance of outliers, there is an emerging need to reassess the fundamentals of current quantization literature. The emphasis should pivot towards optimizing inference speed, rather than primarily focusing on outlier preservation, to align with the evolving characteristics of state-of-the-art LLMs.