MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning

TL;DR

MARBLE is a challenging multimodal reasoning benchmark designed to evaluate the step-by-step reasoning and planning abilities of multimodal language models in complex spatial, visual, and physical environments, revealing significant performance gaps.

Contribution

The paper introduces MARBLE, a novel benchmark with two complex tasks, to assess and highlight the limitations of current multimodal language models in reasoning and planning.

Findings

Current MLLMs perform poorly on MARBLE tasks.

Models achieve near-random performance on complex tasks.

Perception remains a bottleneck in multimodal reasoning.

Abstract

The ability to process information from multiple modalities and to reason through it step-by-step remains a critical challenge in advancing artificial intelligence. However, existing reasoning benchmarks focus on text-only reasoning, or employ multimodal questions that can be answered by directly retrieving information from a non-text modality. Thus, complex reasoning remains poorly understood in multimodal domains. Here, we present MARBLE, a challenging multimodal reasoning benchmark that is designed to scrutinize multimodal language models (MLLMs) in their ability to carefully reason step-by-step through complex multimodal problems and environments. MARBLE is composed of two highly challenging tasks, M-Portal and M-Cube, that require the crafting and understanding of multistep plans under spatial, visual, and physical constraints. We find that current MLLMs perform poorly on MARBLE -- all the 12 advanced models obtain near-random performance on M-Portal and 0% accuracy on M-Cube. Only in simplified subtasks some models outperform the random baseline, indicating that complex reasoning is still a challenge for existing MLLMs. Moreover, we show that perception remains a bottleneck, where MLLMs occasionally fail to extract information from the visual inputs. By shedding a light on the limitations of MLLMs, we hope that MARBLE will spur the development of the next generation of models with the ability to reason and plan across many, multimodal reasoning steps.