Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?

TL;DR

This paper introduces Argus Inspection, a benchmark for evaluating multimodal large language models' visual and causal reasoning, revealing significant room for improvement in their fine-grained perception and commonsense inference abilities.

Contribution

It presents a new benchmark and evaluation framework that specifically assess detailed visual recognition and causal reasoning in MLLMs, highlighting current limitations.

Findings

Maximum visual fine-grained reasoning accuracy is only 0.46.

The Eye of Panoptes framework enables holistic response evaluation.

Most MLLMs still have substantial room for improvement.

Abstract

As Multimodal Large Language Models (MLLMs) continue to evolve, their cognitive and reasoning capabilities have seen remarkable progress. However, challenges in visual fine-grained perception and commonsense causal inference persist. This paper introduces Argus Inspection, a multimodal benchmark with two levels of difficulty, emphasizing detailed visual recognition while incorporating real-world commonsense understanding to evaluate causal reasoning abilities. Expanding on it, we present the Eye of Panoptes framework, which integrates a binary parametric Sigmoid metric with an indicator function, enabling a more holistic evaluation of MLLMs' responses in opinion-based reasoning tasks. Experiments conducted on 26 mainstream MLLMs reveal that the highest performance in visual fine-grained reasoning reaches only 0.46, highlighting considerable potential for enhancement. Our research offers valuable perspectives for the continued refinement of MLLMs.