Learning 6-DoF Task-oriented Grasp Detection via Implicit Estimation and Visual Affordance

TL;DR

This paper introduces a novel 6-DoF task-oriented grasp detection framework that leverages implicit estimation and visual affordance to predict diverse grasp poses directly from point clouds, improving accuracy and applicability in real-world scenarios.

Contribution

It proposes a new grasp detection method using implicit estimation and visual affordance networks, overcoming pixel-level limitations and enabling stable real-world robot grasping.

Findings

Significant improvement over baselines in simulation tests.

Effective transfer from simulated training to real robot execution.

Robust grasp candidate generation for various objects and tasks.

Abstract

Currently, task-oriented grasp detection approaches are mostly based on pixel-level affordance detection and semantic segmentation. These pixel-level approaches heavily rely on the accuracy of a 2D affordance mask, and the generated grasp candidates are restricted to a small workspace. To mitigate these limitations, we first construct a novel affordance-based grasp dataset and propose a 6-DoF task-oriented grasp detection framework, which takes the observed object point cloud as input and predicts diverse 6-DoF grasp poses for different tasks. Specifically, our implicit estimation network and visual affordance network in this framework could directly predict coarse grasp candidates, and corresponding 3D affordance heatmap for each potential task, respectively. Furthermore, the grasping scores from coarse grasps are combined with heatmap values to generate more accurate and finer candidates. Our proposed framework shows significant improvements compared to baselines for existing and novel objects on our simulation dataset. Although our framework is trained based on the simulated objects and environment, the final generated grasp candidates can be accurately and stably executed in real robot experiments when the object is randomly placed on a support surface.