Language-driven Grasp Detection

TL;DR

This paper introduces Grasp-Anything++, a large-scale, language-driven dataset and a diffusion model-based method for grasp detection, enabling robots to interpret natural language instructions for grasping objects in complex scenes.

Contribution

The paper presents a novel large-scale dataset and a diffusion model approach for language-driven grasp detection, incorporating contrastive training for improved performance.

Findings

Outperforms state-of-the-art grasp detection methods

Enables zero-shot grasp detection with natural language

Supports real-world robotic grasping applications

Abstract

Grasp detection is a persistent and intricate challenge with various industrial applications. Recently, many methods and datasets have been proposed to tackle the grasp detection problem. However, most of them do not consider using natural language as a condition to detect the grasp poses. In this paper, we introduce Grasp-Anything++, a new language-driven grasp detection dataset featuring 1M samples, over 3M objects, and upwards of 10M grasping instructions. We utilize foundation models to create a large-scale scene corpus with corresponding images and grasp prompts. We approach the language-driven grasp detection task as a conditional generation problem. Drawing on the success of diffusion models in generative tasks and given that language plays a vital role in this task, we propose a new language-driven grasp detection method based on diffusion models. Our key contribution is the contrastive training objective, which explicitly contributes to the denoising process to detect the grasp pose given the language instructions. We illustrate that our approach is theoretically supportive. The intensive experiments show that our method outperforms state-of-the-art approaches and allows real-world robotic grasping. Finally, we demonstrate our large-scale dataset enables zero-short grasp detection and is a challenging benchmark for future work. Project website: https://airvlab.github.io/grasp-anything/