GraspMolmo: Generalizable Task-Oriented Grasping via Large-Scale Synthetic Data Generation

TL;DR

GraspMolmo is a novel model for task-oriented robotic grasping that leverages a large synthetic dataset to generalize across diverse instructions and objects, achieving state-of-the-art results in complex real-world tasks.

Contribution

The paper introduces GraspMolmo, a new model trained on PRISM, a large-scale synthetic dataset, enabling generalizable, open-vocabulary, task-oriented grasping in cluttered environments.

Findings

Achieves 70% success on complex real-world tasks.

Outperforms previous methods with 35% success rate.

Demonstrates zero-shot semantic grasping capabilities.

Abstract

We present GrasMolmo, a generalizable open-vocabulary task-oriented grasping (TOG) model. GraspMolmo predicts semantically appropriate, stable grasps conditioned on a natural language instruction and a single RGB-D frame. For instance, given "pour me some tea", GraspMolmo selects a grasp on a teapot handle rather than its body. Unlike prior TOG methods, which are limited by small datasets, simplistic language, and uncluttered scenes, GraspMolmo learns from PRISM, a novel large-scale synthetic dataset of 379k samples featuring cluttered environments and diverse, realistic task descriptions. We fine-tune the Molmo visual-language model on this data, enabling GraspMolmo to generalize to novel open-vocabulary instructions and objects. In challenging real-world evaluations, GraspMolmo achieves state-of-the-art results, with a 70% prediction success on complex tasks, compared to the 35% achieved by the next best alternative. GraspMolmo also successfully demonstrates the ability to predict semantically correct bimanual grasps zero-shot. We release our synthetic dataset, code, model, and benchmarks to accelerate research in task-semantic robotic manipulation, which, along with videos, are available at https://abhaybd.github.io/GraspMolmo/.