SEM: Enhancing Spatial Understanding for Robust Robot Manipulation

TL;DR

This paper introduces SEM, a diffusion-based policy framework that improves spatial understanding in robot manipulation by combining 3D geometric context and embodiment-aware modeling, resulting in more robust and generalizable performance.

Contribution

The paper presents SEM, a novel framework integrating a spatial enhancer and robot state encoder to enhance spatial reasoning in manipulation policies.

Findings

SEM outperforms existing baselines in diverse manipulation tasks.

The spatial enhancer improves 3D geometric understanding.

The robot state encoder captures embodiment-aware structures effectively.

Abstract

A key challenge in robot manipulation lies in developing policy models with strong spatial understanding, the ability to reason about 3D geometry, object relations, and robot embodiment. Existing methods often fall short: 3D point cloud models lack semantic abstraction, while 2D image encoders struggle with spatial reasoning. To address this, we propose SEM (Spatial Enhanced Manipulation model), a novel diffusion-based policy framework that explicitly enhances spatial understanding from two complementary perspectives. A spatial enhancer augments visual representations with 3D geometric context, while a robot state encoder captures embodiment-aware structure through graphbased modeling of joint dependencies. By integrating these modules, SEM significantly improves spatial understanding, leading to robust and generalizable manipulation across diverse tasks that outperform existing baselines.