VLMQ: Token Saliency-Driven Post-Training Quantization for Vision-language Models

TL;DR

VLMQ introduces a token saliency-driven post-training quantization method tailored for vision-language models, effectively addressing visual over-representation and modality gap issues to improve quantization performance, especially in low-bit settings.

Contribution

The paper proposes VLMQ, a novel PTQ framework that prioritizes salient tokens using gradient-based importance, with lightweight backpropagation and importance-aware optimization, specifically designed for VLMs.

Findings

Achieves state-of-the-art quantization performance on 8 benchmarks.

Demonstrates 16.45% improvement on MME-RealWorld under 2-bit quantization.

Effectively addresses visual over-representation and modality gap in VLMs.

Abstract

Post-training quantization (PTQ) has emerged as an effective technique for compressing large models and accelerating inference without retraining. While PTQ has been extensively studied in large language models (LLMs), its application to vision-language models (VLMs) remains underexplored. In this work, we identify two intrinsic characteristics of VLM activations: 1) visual over-representation, where vision tokens are excessive and often redundant, and 2) modality gap, which refers to the clear distribution gap between text and vision tokens in the latent feature space. Together, these two factors significantly deteriorate quantization performance but have been overlooked by existing PTQ methods. To address these challenges, we propose VLMQ, A VLM-tailored PTQ framework that selectively prioritizes salient tokens while suppressing redundant ones during quantization. In particular, we introduce a gradient-driven importance factor to capture the token-wise importance variance, the effectiveness of which is substantiated through both empirical and theoretical analysis. To ensure efficiency, we propose to use lightweight block-wise backpropagation for factor acquisition. Finally, we reformulate the optimization objective into an importance-aware form to preserve important activation information. Extensive evaluations on 8 benchmarks across 0.5B$\sim$32B VLMs demonstrate the state-of-the-art (SOTA) performance of our VLMQ, particularly under low-bit settings. For example, it achieves a substantial \textbf{16.45\%} improvement on MME-RealWorld under 2-bit quantization.