RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping

TL;DR

RAGNet introduces a large-scale, reasoning-based affordance segmentation benchmark with diverse data and instructions, enabling improved generalization for robotic grasping in open-world scenarios.

Contribution

It provides a comprehensive large-scale dataset and a novel affordance-based grasping framework, AffordanceNet, enhancing open-world robotic grasping capabilities.

Findings

Model demonstrates strong generalization in open-world scenarios.

Benchmark covers diverse data domains and complex instructions.

AffordanceNet outperforms existing methods in grasping tasks.

Abstract

General robotic grasping systems require accurate object affordance perception in diverse open-world scenarios following human instructions. However, current studies suffer from the problem of lacking reasoning-based large-scale affordance prediction data, leading to considerable concern about open-world effectiveness. To address this limitation, we build a large-scale grasping-oriented affordance segmentation benchmark with human-like instructions, named RAGNet. It contains 273k images, 180 categories, and 26k reasoning instructions. The images cover diverse embodied data domains, such as wild, robot, ego-centric, and even simulation data. They are carefully annotated with an affordance map, while the difficulty of language instructions is largely increased by removing their category name and only providing functional descriptions. Furthermore, we propose a comprehensive affordance-based grasping framework, named AffordanceNet, which consists of a VLM pre-trained on our massive affordance data and a grasping network that conditions an affordance map to grasp the target. Extensive experiments on affordance segmentation benchmarks and real-robot manipulation tasks show that our model has a powerful open-world generalization ability. Our data and code is available at https://github.com/wudongming97/AffordanceNet.