Does Visual Token Pruning Improve Calibration? An Empirical Study on Confidence in MLLMs

TL;DR

This study investigates how visual token pruning impacts the calibration of confidence estimates in multimodal large language models, revealing that pruning can improve confidence reliability without sacrificing accuracy.

Contribution

It provides the first comprehensive empirical analysis of the effects of visual token pruning on model calibration in MLLMs, highlighting potential calibration benefits.

Findings

Pruning can lower Expected Calibration Error (ECE) while maintaining accuracy.

Reducing saliency weight in pruning improves calibration across token budgets.

Saliency-based pruning may worsen calibration, and FastV causes performance degradation.

Abstract

Visual token pruning is a widely used strategy for efficient inference in multimodal large language models (MLLMs), but existing work mainly evaluates it with task accuracy. In this paper, we study how visual token pruning affects model calibration, that is, whether predicted confidence matches actual correctness. Using LLaVA-1.5-7B on POPE and ScienceQA-IMG, we evaluate Expected Calibration Error (ECE), Brier score, and AURC under several pruning strategies, including SCOPE with different saliency weights, saliency-only pruning, FastV, and random pruning, across multiple token budgets. Our results show that pruning does not simply trade reliability for efficiency. On POPE, a pure-coverage setting in SCOPE achieves substantially lower ECE than the full unpruned model while maintaining similar accuracy. An internal alpha-sweep further shows a consistent trend: reducing the saliency weight improves calibration at all tested token budgets, while accuracy changes only slightly. In contrast, saliency-based pruning leads to worse calibration, and real FastV causes severe performance degradation in our setting. On ScienceQA-IMG, pruning also reduces ECE, with accuracy remaining stable or slightly improving. We additionally study the gap power exponent in coverage-based selection and find that its default setting is not always optimal. Overall, our results suggest that visual token pruning should be evaluated not only by accuracy, but also by confidence quality, especially for multimodal systems that need reliable decisions.