FoundationGrasp: Generalizable Task-Oriented Grasping with Foundation Models

TL;DR

FoundationGrasp introduces a foundation model-based framework for task-oriented grasping that generalizes well to new objects and tasks, validated through extensive experiments and real-robot tests.

Contribution

It leverages foundation models to enable open-ended, generalizable task-oriented grasping, surpassing prior methods limited to closed-set knowledge.

Findings

Outperforms existing methods on LaViA-TaskGrasp dataset

Successfully generalizes to novel objects and tasks

Validated in real-robot grasping experiments

Abstract

Task-oriented grasping (TOG), which refers to synthesizing grasps on an object that are configurationally compatible with the downstream manipulation task, is the first milestone towards tool manipulation. Analogous to the activation of two brain regions responsible for semantic and geometric reasoning during cognitive processes, modeling the intricate relationship between objects, tasks, and grasps necessitates rich semantic and geometric prior knowledge about these elements. Existing methods typically restrict the prior knowledge to a closed-set scope, limiting their generalization to novel objects and tasks out of the training set. To address such a limitation, we propose FoundationGrasp, a foundation model-based TOG framework that leverages the open-ended knowledge from foundation models to learn generalizable TOG skills. Extensive experiments are conducted on the contributed Language and Vision Augmented TaskGrasp (LaViA-TaskGrasp) dataset, demonstrating the superiority of FoundationGrasp over existing methods when generalizing to novel object instances, object classes, and tasks out of the training set. Furthermore, the effectiveness of FoundationGrasp is validated in real-robot grasping and manipulation experiments on a 7-DoF robotic arm. Our code, data, appendix, and video are publicly available at https://sites.google.com/view/foundationgrasp.