Multimodal Large Language Models as Image Classifiers

TL;DR

This paper critically examines evaluation protocols for Multimodal Large Language Models (MLLMs), identifies key issues affecting performance assessment, and demonstrates that corrected evaluation and label quality significantly improve perceived accuracy and utility in image classification tasks.

Contribution

The paper identifies flaws in current evaluation protocols for MLLMs, proposes fixes, and shows that improved evaluation and label quality reveal better performance and potential applications.

Findings

Corrected evaluation protocols increase MLLM accuracy by up to 10.8%.

Flawed evaluation protocols have led to underestimating MLLMs' true capabilities.

MLLMs can effectively assist human annotators in dataset curation.

Abstract

Multimodal Large Language Models (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies comparing MLLMs with supervised and vision-language models report conflicting conclusions, and we show these conflicts stem from protocols that either inflate or underestimate performance. Across the most common evaluation protocols, we identify and fix key issues: model outputs that fall outside the provided class list and are discarded, inflated results from weak multiple-choice distractors, and an open-world setting that underperforms only due to poor output mapping. We additionally quantify the impact of commonly overlooked design choices - batch size, image ordering, and text encoder selection - showing they substantially affect accuracy. Evaluating on ReGT, our multilabel reannotation of 625 ImageNet-1k classes, reveals that MLLMs benefit most from corrected labels (up to +10.8%), substantially narrowing the perceived gap with supervised models. Much of the reported MLLMs underperformance on classification is thus an artifact of noisy ground truth and flawed evaluation protocol rather than genuine model deficiency. Models less reliant on supervised training signals prove most sensitive to annotation quality. Finally, we show that MLLMs can assist human annotators: in a controlled case study, annotators confirmed or integrated MLLMs predictions in approximately 50% of difficult cases, demonstrating their potential for large-scale dataset curation. This work is part of the Aiming for Perfect ImageNet-1k project, see https://klarajanouskova.github.io/ImageNet/.