pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning

TL;DR

pySpatial enables large language models to perform zero-shot 3D spatial reasoning by generating Python code that interfaces with spatial tools, transforming 2D inputs into 3D scenes for explicit reasoning.

Contribution

We introduce pySpatial, a zero-shot visual programming framework that enhances MLLMs with 3D spatial reasoning capabilities without fine-tuning.

Findings

Outperforms baseline MLLMs on MindCube and Omni3D-Bench benchmarks.

Achieves 12.94% higher accuracy than GPT-4.1-mini on MindCube.

Enables real-world indoor navigation using generated route plans.

Abstract

Multi-modal Large Language Models (MLLMs) have demonstrated strong capabilities in general-purpose perception and reasoning, but they still struggle with tasks that require spatial understanding of the 3D world. To address this, we introduce pySpatial, a visual programming framework that equips MLLMs with the ability to interface with spatial tools via Python code generation. Given an image sequence and a natural-language query, the model composes function calls to spatial tools including 3D reconstruction, camera-pose recovery, novel-view rendering, etc. These operations convert raw 2D inputs into an explorable 3D scene, enabling MLLMs to reason explicitly over structured spatial representations. Notably, pySpatial requires no gradient-based fine-tuning and operates in a fully zero-shot setting. Experimental evaluations on the challenging MindCube and Omni3D-Bench benchmarks demonstrate that our framework pySpatial consistently surpasses strong MLLM baselines; for instance, it outperforms GPT-4.1-mini by 12.94% on MindCube. Furthermore, we conduct real-world indoor navigation experiments where the robot can successfully traverse complex environments using route plans generated by pySpatial, highlighting the practical effectiveness of our approach.