3D Affordance Keypoint Detection for Robotic Manipulation

TL;DR

This paper introduces a novel 3D keypoint detection method for robotic manipulation that improves understanding of object functionality and guides manipulation, outperforming existing models in segmentation and keypoint detection tasks.

Contribution

The paper proposes FAKP-Net, a fusion-based network utilizing 3D keypoints and RGB-D data to enhance affordance detection and manipulation guidance.

Findings

FAKP-Net outperforms existing models in benchmark tests.

The method reliably guides manipulation of unseen objects.

Real-world experiments validate the approach's effectiveness.

Abstract

This paper presents a novel approach for affordance-informed robotic manipulation by introducing 3D keypoints to enhance the understanding of object parts' functionality. The proposed approach provides direct information about what the potential use of objects is, as well as guidance on where and how a manipulator should engage, whereas conventional methods treat affordance detection as a semantic segmentation task, focusing solely on answering the what question. To address this gap, we propose a Fusion-based Affordance Keypoint Network (FAKP-Net) by introducing 3D keypoint quadruplet that harnesses the synergistic potential of RGB and Depth image to provide information on execution position, direction, and extent. Benchmark testing demonstrates that FAKP-Net outperforms existing models by significant margins in affordance segmentation task and keypoint detection task. Real-world experiments also showcase the reliability of our method in accomplishing manipulation tasks with previously unseen objects.