Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage Task

TL;DR

This paper introduces MultiStAR, a multi-stage benchmark for abstract visual reasoning, and MSEval, a metric that evaluates both intermediate reasoning steps and final outcomes in multimodal large language models.

Contribution

The paper presents a new multi-stage AVR benchmark and a novel metric to better evaluate reasoning processes in MLLMs, addressing limitations of existing single-step benchmarks and metrics.

Findings

MLLMs perform well on perception tasks

MLLMs struggle with complex rule detection

Intermediate reasoning remains challenging

Abstract

Current Multimodal Large Language Models (MLLMs) excel in general visual reasoning but remain underexplored in Abstract Visual Reasoning (AVR), which demands higher-order reasoning to identify abstract rules beyond simple perception. Existing AVR benchmarks focus on single-step reasoning, emphasizing the end result but neglecting the multi-stage nature of reasoning process. Past studies found MLLMs struggle with these benchmarks, but it doesn't explain how they fail. To address this gap, we introduce MultiStAR, a Multi-Stage AVR benchmark, based on RAVEN, designed to assess reasoning across varying levels of complexity. Additionally, existing metrics like accuracy only focus on the final outcomes while do not account for the correctness of intermediate steps. Therefore, we propose a novel metric, MSEval, which considers the correctness of intermediate steps in addition to the final outcomes. We conduct comprehensive experiments on MultiStAR using 17 representative close-source and open-source MLLMs. The results reveal that while existing MLLMs perform adequately on basic perception tasks, they continue to face challenges in more complex rule detection stages.