Learning Spatial-Aware Manipulation Ordering

TL;DR

OrderMind is a novel framework that learns spatial-aware manipulation sequences in cluttered environments, improving the efficiency and robustness of robotic manipulation through spatial context encoding and supervision from a vision-language model.

Contribution

We introduce OrderMind, a unified spatial-aware manipulation ordering framework that leverages spatial context encoding and a spatial prior labeling method for improved manipulation planning.

Findings

Outperforms prior methods in effectiveness and efficiency

Successfully applied in both simulation and real-world environments

Handles complex cluttered scenes with high accuracy

Abstract

Manipulation in cluttered environments is challenging due to spatial dependencies among objects, where an improper manipulation order can cause collisions or blocked access. Existing approaches often overlook these spatial relationships, limiting their flexibility and scalability. To address these limitations, we propose OrderMind, a unified spatial-aware manipulation ordering framework that directly learns object manipulation priorities based on spatial context. Our architecture integrates a spatial context encoder with a temporal priority structuring module. We construct a spatial graph using k-Nearest Neighbors to aggregate geometric information from the local layout and encode both object-object and object-manipulator interactions to support accurate manipulation ordering in real-time. To generate physically and semantically plausible supervision signals, we introduce a spatial prior labeling method that guides a vision-language model to produce reasonable manipulation orders for distillation. We evaluate OrderMind on our Manipulation Ordering Benchmark, comprising 163,222 samples of varying difficulty. Extensive experiments in both simulation and real-world environments demonstrate that our method significantly outperforms prior approaches in effectiveness and efficiency, enabling robust manipulation in cluttered scenes.