AssemLM: Spatial Reasoning Multimodal Large Language Models for Robotic Assembly

TL;DR

AssemLM is a multimodal large language model designed for robotic assembly, integrating 3D perception and reasoning to predict precise 6D poses, supported by a new large-scale dataset for spatial reasoning.

Contribution

The paper introduces AssemLM, a novel spatial multimodal LLM for robotic assembly, and presents AssemBench, a large dataset for 3D spatial reasoning evaluation.

Findings

Achieves state-of-the-art 6D pose reasoning accuracy.

Supports fine-grained, multi-step robotic assembly in real-world tests.

Extends spatial reasoning benchmarks into full 3D geometric inference.

Abstract

Spatial reasoning is a fundamental capability for embodied intelligence, especially for fine-grained manipulation tasks such as robotic assembly. While recent vision-language models (VLMs) exhibit preliminary spatial awareness, they largely rely on coarse 2D perception and lack the ability to perform accurate reasoning over 3D geometry, which is crucial for precise assembly operations. To address this limitation, we propose AssemLM, a spatial multimodal large language model tailored for robotic assembly. AssemLM integrates assembly manuals, point clouds, and textual instructions to reason about and predict task-critical 6D assembly poses, enabling explicit geometric understanding throughout the assembly process. To effectively bridge raw 3D perception and high-level reasoning, we adopt a specialized point cloud encoder to capture fine-grained geometric and rotational features, which are then integrated into the multimodal language model to support accurate 3D spatial reasoning for assembly tasks. In addition, we construct AssemBench, a large-scale dataset and benchmark for assembly-oriented spatial reasoning, comprising over 900K multimodal samples with precise 6D pose annotations. AssemBench extends spatial reasoning evaluation beyond 2D and grounding tasks into full 3D geometric inference, filling a critical gap in existing embodied AI benchmarks. Extensive experiments demonstrate that AssemLM achieves state-of-the-art performance in 6D pose reasoning across diverse assembly scenarios. Furthermore, real-robot evaluations show that our model can support fine-grained and multi-step assembly execution in real-world settings, demonstrating its potential for robotic assembly applications.