Evaluating the Impact of Post-Training Quantization on Reliable VQA with Multimodal LLMs

TL;DR

This paper systematically examines how post-training quantization affects the accuracy and reliability of multimodal large language models in visual question answering, proposing methods to mitigate reliability loss and optimize efficiency.

Contribution

It is the first systematic study linking quantization effects to reliability in multimodal models, introducing a robustness-adapted confidence estimator and demonstrating effective quantization strategies.

Findings

PTQ degrades accuracy and reliability in VQA models.

Data-aware quantization methods mitigate some reliability loss.

Combining int4 MBQ with the Selector achieves near-uncompressed performance with 75% less memory.

Abstract

Multimodal Large Language Models (MLLM) are increasingly deployed in domains where both reliability and efficiency are critical. However, current models remain overconfident, producing highly certain but incorrect answers. At the same time, their large size limits deployment on edge devices, necessitating compression. We study the intersection of these two challenges by analyzing how Post-Training Quantization (PTQ) compression affects both accuracy and reliability in Visual Question Answering (VQA). We evaluate two MLLMs, Qwen2-VL-7B and Idefics3-8B, quantized with data-free (HQQ) and data-aware (MBQ) methods across multiple bit widths. To counteract the reduction in reliability caused by quantization, we adapt the Selector confidence estimator for quantized multimodal settings and test its robustness across various quantization levels and out-of-distribution (OOD) scenarios. We find that PTQ degrades both accuracy and reliability. Data-aware methods soften the effect thereof. The Selector substantially mitigates the reliability impact. The combination of int4 MBQ and the Selector achieves the best efficiency-reliability trade-off, closing in on uncompressed performance at approx. 75% less memory demand. Overall, we present the first systematic study linking quantization and reliability in multimodal settings.