VEQ: Modality-Adaptive Quantization for MoE Vision-Language Models

TL;DR

VEQ introduces a modality-adaptive quantization framework for MoE vision-language models, effectively reducing memory and computation costs while maintaining high accuracy by addressing cross-modal and expert heterogeneity.

Contribution

The paper proposes VEQ, a novel dual-aware quantization method that considers modality differences and expert heterogeneity, improving compression performance of MoE VLMs.

Findings

VEQ outperforms state-of-the-art quantization methods on multiple benchmarks.

Achieves over 2% accuracy improvement on Kimi-VL and Qwen3-VL.

Demonstrates robustness across various multimodal tasks.

Abstract

Mixture-of-Experts(MoE) Vision-Language Models (VLMs) offer remarkable performance but incur prohibitive memory and computational costs, making compression essential. Post-Training Quantization (PTQ) is an effective training-free technique to address the massive memory and computation overhead. Existing quantization paradigms fall short as they are oblivious to two critical forms of heterogeneity: the inherent discrepancy between vision and language tokens, and the non-uniform contribution of different experts. To bridge this gap, we propose Visual Expert Quantization (VEQ), a dual-aware quantization framework designed to simultaneously accommodate cross-modal differences and heterogeneity between experts. Specifically, VEQ incorporates 1)Modality-expert-aware Quantization, which utilizes expert activation frequency to prioritize error minimization for pivotal experts, and 2)Modality-affinity-aware Quantization, which constructs an enhanced Hessian matrix by integrating token-expert affinity with modality information to guide the calibration process. Extensive experiments across diverse benchmarks verify that VEQ consistently outperforms state-of-the-art baselines. Specifically, under the W3A16 configuration, our method achieves significant average accuracy gains of 2.04\% on Kimi-VL and 3.09\% on Qwen3-VL compared to the previous SOTA quantization methods, demonstrating superior robustness across various multimodal tasks. Our code will be available at https://github.com/guangshuoqin/VEQ.