MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI

TL;DR

MMReason is a comprehensive benchmark designed to evaluate long-chain reasoning in multimodal large language models across diverse, challenging, and open-ended questions, addressing limitations of existing benchmarks.

Contribution

The paper introduces MMReason, a new benchmark with diverse, multi-disciplinary, and difficulty-scaled questions, along with a multi-model filtering and detailed step-by-step annotation for robust reasoning assessment.

Findings

Benchmarking reveals current MLLMs have limited reasoning capabilities.

The dataset's multi-step questions challenge models beyond simple pattern recognition.

The scoring mechanism provides reliable assessment of intermediate reasoning steps.

Abstract

Reasoning plays a crucial role in advancing Multimodal Large Language Models (MLLMs) toward Artificial General Intelligence. However, existing MLLM benchmarks often fall short in precisely and comprehensively evaluating long-chain reasoning abilities from three key aspects: (1) lack of difficulty and diversity, (2) susceptibility to guessability and memorization, (3) inadequate assessment of intermediate reasoning steps. To fill this gap, we introduce MMReason, a new benchmark designed to precisely and comprehensively evaluate MLLM long-chain reasoning capability with diverse, open-ended, challenging questions. First, we curate challenging questions requiring multi-step reasoning from various fields (i.e., 6 disciplines) and multiple difficulty levels (i.e., from pre-university to university, and from foundational to competition tiers). Second, these questions are reformulated into an open-ended format and filtered using a multi-model voting technique to eliminate shortcut cases related to guessing and memorization, ensuring robust reasoning evaluations. Third, we annotate the questions with detailed step-by-step solutions, and design a reference-based ternary scoring mechanism to reliably assess intermediate reasoning steps. With MMReason, we benchmark popular leading MLLMs and provide an in-depth analysis of their reasoning capabilities. We hope MMReason will serve as a valuable resource for advancing MLLM reasoning research. Code will be available at https://github.com/HJYao00/MMReason.