Rethinking Practical and Efficient Quantization Calibration for Vision-Language Models

TL;DR

This paper introduces TLQ, a novel token-level, layer-wise quantization calibration method for vision-language models that improves performance and stability by leveraging gradient-based importance and multi-GPU calibration schemes.

Contribution

It proposes a new PTQ calibration framework for VLMs that uses token-level importance and multi-GPU calibration to enhance quantization accuracy and efficiency.

Findings

Consistently improves quantized model performance across various models and settings.

Reduces GPU memory requirements for calibration, enabling scalable deployment.

Achieves stable and robust quantization results in vision-language models.

Abstract

Post-training quantization (PTQ) is a primary approach for deploying large language models without fine-tuning, and the quantized performance is often strongly affected by the calibration in PTQ. By contrast, in vision-language models (VLMs), substantial differences between visual and text tokens in their activation distributions and sensitivities to quantization error pose significant challenges for effective calibration during PTQ. In this work, we rethink what PTQ calibration should align with in VLMs and propose the Token-level Importance-aware Layer-wise Quantization framework (TLQ). Guided by gradient information, we design a token-level importance integration mechanism for quantization error, and use it to construct a token-level calibration set, enabling a more fine-grained calibration strategy. Furthermore, TLQ introduces a multi-GPU, quantization-exposed layer-wise calibration scheme. This scheme keeps the layer-wise calibration procedure consistent with the true quantized inference path and distributes the complex layer-wise calibration workload across multiple RTX3090 GPUs, thereby reducing reliance on the large memory of A100 GPUs. TLQ is evaluated across two models, three model scales, and two quantization settings, consistently achieving performance improvements across all settings, indicating its strong quantization stability. The code will be released publicly.