LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models

TL;DR

LAST is a framework that improves spatial reasoning in multimodal large language models by integrating specialized vision tools and a progressive training strategy to better understand and utilize diverse spatial cues.

Contribution

The paper introduces LAST, a unified tool-augmented spatial reasoning framework with an interactive sandbox and training strategy, addressing challenges in heterogeneous tool invocation and output understanding.

Findings

LAST-7B achieves 20% performance gains over its backbone.

Outperforms strong proprietary closed-source LLMs.

Enhances reasoning on complex spatial tasks.

Abstract

Spatial reasoning is a cornerstone capability for intelligent systems to perceive and interact with the physical world. However, multimodal large language models (MLLMs) frequently suffer from hallucinations and imprecision when parsing complex geometric layouts. As data-driven scaling struggles to internalize structured geometric priors and spatial constraints, integrating mature, specialized vision models presents a compelling alternative. Despite its promise, applying this paradigm to spatial reasoning is hindered by two key challenges: The difficulty of invoking heterogeneous, parameter-rich tools, as well as the challenge of understanding and effectively leveraging their diverse low-level outputs (e.g., segmentation masks, depth maps) in high-level reasoning. To address these challenges, we propose LAST, a unified framework for tool-augmented spatial reasoning. LAST features an extensible interactive sandbox, termed LAST-Box, which abstracts heterogeneous tool invocations into atomic instructions and reusable spatial skills, returning multimodal hints (e.g., annotated images and textual descriptions) that can be directly consumed by LLMs. We further design a three-stage progressive training strategy that guides models from understanding tool outputs to proficient and adaptive tool invocation. Experiments on four datasets show that LAST-7B achieves around 20\% performance gains over its backbone and outperforms strong proprietary closed-source LLMs, substantially enhancing reasoning on complex spatial tasks.