MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization

TL;DR

MQuant is a novel post-training quantization framework that significantly reduces the inference latency of multimodal large language models while maintaining near-floating-point accuracy, enabling more practical deployment on resource-limited devices.

Contribution

MQuant introduces modality-specific static quantization, attention-invariant flexible switching, and rotation magnitude suppression to address unique challenges in quantizing MLLMs.

Findings

Achieves <1% accuracy degradation with W4A8 quantization.

Reduces inference latency by up to 30%.

Outperforms existing PTQ baselines on five mainstream MLLMs.

Abstract

Multimodal large language models (MLLMs) have garnered widespread attention due to their ability to understand multimodal input. However, their large parameter sizes and substantial computational demands severely hinder their practical deployment and application.While quantization is an effective way to reduce model size and inference latency, its application to MLLMs remains underexplored. In this paper, we propose MQuant, a post-training quantization (PTQ) framework designed to tackle the unique challenges of multimodal large language models (MLLMs). Conventional quantization often struggles with MLLMs because of (a) high inference latency from large visual token counts, (b) distributional disparities between visual and textual tokens, and (c) extreme outliers introduced by Hadamard-based transformations. To address these issues, MQuant introduces: Modality-Specific Static Quantization (MSQ), assigning distinct static scales for visual vs. textual tokens; Attention-Invariant Flexible Switching (AIFS), reordering tokens to preserve casual attention while eliminating expensive token-wise scale computations; Rotation Magnitude Suppression (RMS), mitigating weight outliers arising from online Hadamard rotations. On five mainstream MLLMs (including Qwen-VL, MiniCPM-V, CogVLM2), MQuant under W4A8 achieves near-floating-point accuracy (<1% degradation) while reducing inference latency by up to 30%, significantly outperforming existing PTQ baselines. Our MQuant effectively bridges the gap for efficient and accurate MLLMs inference in resource-constrained devices. Code has been released in https://github.com/StiphyJay/MQuant.