On the Importance of a Multi-Scale Calibration for Quantization

TL;DR

This paper introduces MaCa, a multi-scale calibration method that improves post-training quantization of large language models by accounting for variable input lengths, leading to more accurate Hessian estimates and better model performance.

Contribution

MaCa is the first method to incorporate multi-scale sequence length information into Hessian estimation for LLM quantization, enhancing accuracy in low-bit settings.

Findings

MaCa consistently improves quantization accuracy on LLMs like Qwen3, Gemma3, and LLaMA3.

MaCa offers a lightweight, compatible enhancement for existing PTQ frameworks.

Multi-scale calibration significantly impacts Hessian-based weight importance estimation.

Abstract

Post-training quantization (PTQ) is a cornerstone for efficiently deploying large language models (LLMs), where a small calibration set critically affects quantization performance. However, conventional practices rely on random sequences of fixed length, overlooking the variable-length nature of LLM inputs. Input length directly influences the activation distribution and, consequently, the weight importance captured by the Hessian, which in turn affects quantization outcomes. As a result, Hessian estimates derived from fixed-length calibration may fail to represent the true importance of weights across diverse input scenarios. We propose MaCa (Matryoshka Calibration), a simple yet effective method for length-aware Hessian construction. MaCa (i) incorporates multi-scale sequence length information into Hessian estimation and (ii) regularizes each sequence as an independent sample, yielding a more stable and fruitful Hessian for accurate quantization. Experiments on state-of-the-art LLMs (e.g., Qwen3, Gemma3, LLaMA3) demonstrate that MaCa consistently improves accuracy under low bit quantization, offering a lightweight enhancement compatible with existing PTQ frameworks. To the best of our knowledge, this is the first work to systematically highlight the role of multi-scale calibration in LLM quantization.