A Modern Take on Visual Relationship Reasoning for Grasp Planning

TL;DR

This paper introduces a transformer-based model for visual relationship reasoning in robotic grasp planning, handling complex scenes with multiple objects, and establishes a new state-of-the-art benchmark with a novel dataset and evaluation metric.

Contribution

It presents D3G, an end-to-end transformer model for detecting objects and their spatial relationships, and introduces D3GD, a new dataset for cluttered bin picking scenes.

Findings

Achieved state-of-the-art performance on relationship detection

Introduced the Average Precision of Relationships metric

Provided a new dataset with diverse cluttered scenes

Abstract

Interacting with real-world cluttered scenes pose several challenges to robotic agents that need to understand complex spatial dependencies among the observed objects to determine optimal pick sequences or efficient object retrieval strategies. Existing solutions typically manage simplified scenarios and focus on predicting pairwise object relationships following an initial object detection phase, but often overlook the global context or struggle with handling redundant and missing object relations. In this work, we present a modern take on visual relational reasoning for grasp planning. We introduce D3GD, a novel testbed that includes bin picking scenes with up to 35 objects from 97 distinct categories. Additionally, we propose D3G, a new end-to-end transformer-based dependency graph generation model that simultaneously detects objects and produces an adjacency matrix representing their spatial relationships. Recognizing the limitations of standard metrics, we employ the Average Precision of Relationships for the first time to evaluate model performance, conducting an extensive experimental benchmark. The obtained results establish our approach as the new state-of-the-art for this task, laying the foundation for future research in robotic manipulation. We publicly release the code and dataset at https://paolotron.github.io/d3g.github.io.