Target-Oriented Object Grasping via Multimodal Human Guidance

TL;DR

This paper introduces TOGNet, a target-oriented grasp detection system that efficiently predicts grasps based on multimodal human guidance, improving success rates in cluttered scenes for robotic grasping tasks.

Contribution

The paper presents a novel target-oriented 6-DoF grasp detection network that integrates multimodal human guidance for more efficient and accurate robotic grasping.

Findings

Achieved a 13.7% success rate improvement in simulation

Demonstrated effectiveness in real-world target-oriented grasping scenarios

Efficiently predicts grasps using local region patches around the target

Abstract

In the context of human-robot interaction and collaboration scenarios, robotic grasping still encounters numerous challenges. Traditional grasp detection methods generally analyze the entire scene to predict grasps, leading to redundancy and inefficiency. In this work, we reconsider 6-DoF grasp detection from a target-referenced perspective and propose a Target-Oriented Grasp Network (TOGNet). TOGNet specifically targets local, object-agnostic region patches to predict grasps more efficiently. It integrates seamlessly with multimodal human guidance, including language instructions, pointing gestures, and interactive clicks. Thus our system comprises two primary functional modules: a guidance module that identifies the target object in 3D space and TOGNet, which detects region-focal 6-DoF grasps around the target, facilitating subsequent motion planning. Through 50 target-grasping simulation experiments in cluttered scenes, our system achieves a success rate improvement of about 13.7%. In real-world experiments, we demonstrate that our method excels in various target-oriented grasping scenarios.