MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models

TL;DR

MASQuant introduces a modality-aware quantization framework for multimodal large language models, effectively addressing cross-modal invariance issues and improving post-training quantization stability and performance.

Contribution

It proposes a novel framework with modality-specific smoothing and cross-modal compensation, advancing quantization techniques for multimodal large language models.

Findings

MASQuant achieves stable quantization across dual- and tri-modal MLLMs.

It outperforms or matches state-of-the-art PTQ algorithms.

The method effectively addresses smoothing misalignment and cross-modal invariance issues.

Abstract

Post-training quantization (PTQ) with computational invariance for Large Language Models~(LLMs) have demonstrated remarkable advances, however, their application to Multimodal Large Language Models~(MLLMs) presents substantial challenges. In this paper, we analyze SmoothQuant as a case study and identify two critical issues: Smoothing Misalignment and Cross-Modal Computational Invariance. To address these issues, we propose Modality-Aware Smoothing Quantization (MASQuant), a novel framework that introduces (1) Modality-Aware Smoothing (MAS), which learns separate, modality-specific smoothing factors to prevent Smoothing Misalignment, and (2) Cross-Modal Compensation (CMC), which addresses Cross-modal Computational Invariance by using SVD whitening to transform multi-modal activation differences into low-rank forms, enabling unified quantization across modalities. MASQuant demonstrates stable quantization performance across both dual-modal and tri-modal MLLMs. Experimental results show that MASQuant is competitive among the state-of-the-art PTQ algorithms. Source code: https://github.com/alibaba/EfficientAI.