ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraints

TL;DR

ORIGAMISPACE introduces a comprehensive benchmark with a new dataset for evaluating multimodal large language models on multi-step spatial reasoning and mathematical constraints using origami tasks.

Contribution

This paper presents ORIGAMISPACE, a novel dataset and benchmark for assessing MLLMs' spatial reasoning and mathematical constraint handling through origami-based tasks.

Findings

Existing MLLMs show strengths in simple tasks but struggle with complex spatial reasoning.

The benchmark reveals specific weaknesses in multi-step reasoning capabilities.

Reinforcement learning shows potential in improving CP code generation.

Abstract

Spatial reasoning is a key capability in the field of artificial intelligence, especially crucial in areas such as robotics, computer vision, and natural language understanding. However, evaluating the ability of multimodal large language models(MLLMs) in complex spatial reasoning still faces challenges, particularly in scenarios requiring multi-step reasoning and precise mathematical constraints. This paper introduces ORIGAMISPACE, a new dataset and benchmark designed to evaluate the multi-step spatial reasoning ability and the capacity to handle mathematical constraints of MLLMs through origami tasks. The dataset contains 350 data instances,each comprising a strictly formatted crease pattern (CP diagram), the Compiled Flat Pattern, the complete Folding Process, and the final Folded Shape Image. We propose four evaluation tasks: Pattern Prediction, Multi-step Spatial Reasoning, Spatial Relationship Prediction, and End-to-End CP Code Generation. For the CP code generation task, we design an interactive environment and explore the possibility of using reinforcement learning methods to train MLLMs. Through experiments on existing MLLMs, we initially reveal the strengths and weaknesses of these models in handling complex spatial reasoning tasks.