GeoManip: Geometric Constraints as General Interfaces for Robot Manipulation

TL;DR

GeoManip is a training-free framework that uses geometric constraints derived from object relationships to enable generalist robots to perform diverse manipulation tasks with high generalization and human interaction capabilities.

Contribution

It introduces a novel, training-free approach leveraging large foundational models to interpret geometric constraints for robotic manipulation, enhancing generalization and human-robot interaction.

Findings

State-of-the-art performance in simulations and real-world tasks

Superior out-of-distribution generalization

Effective human-robot interaction features

Abstract

We present GeoManip, a framework to enable generalist robots to leverage essential conditions derived from object and part relationships, as geometric constraints, for robot manipulation. For example, cutting the carrot requires adhering to a geometric constraint: the blade of the knife should be perpendicular to the carrot's direction. By interpreting these constraints through symbolic language representations and translating them into low-level actions, GeoManip bridges the gap between natural language and robotic execution, enabling greater generalizability across diverse even unseen tasks, objects, and scenarios. Unlike vision-language-action models that require extensive training, operates training-free by utilizing large foundational models: a constraint generation module that predicts stage-specific geometric constraints and a geometry parser that identifies object parts involved in these constraints. A solver then optimizes trajectories to satisfy inferred constraints from task descriptions and the scene. Furthermore, GeoManip learns in-context and provides five appealing human-robot interaction features: on-the-fly policy adaptation, learning from human demonstrations, learning from failure cases, long-horizon action planning, and efficient data collection for imitation learning. Extensive evaluations on both simulations and real-world scenarios demonstrate GeoManip's state-of-the-art performance, with superior out-of-distribution generalization while avoiding costly model training.