I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches

TL;DR

This paper introduces a method for generating robotic grasp configurations based on freehand sketches, leveraging graph structures to handle sketch ambiguity and enabling effective grasp detection with minimal training data.

Contribution

It presents a novel approach that uses graph-structured sketches to improve grasp detection, requiring only around 100 samples for training and enabling real-world application.

Findings

Effective grasp detection from sketches demonstrated on multiple datasets.

Model generalizes well to unseen objects and cluttered scenes.

Physical robot experiments validate practical utility.

Abstract

In this paper, we are interested in the problem of generating target grasps by understanding freehand sketches. The sketch is useful for the persons who cannot formulate language and the cases where a textual description is not available on the fly. However, very few works are aware of the usability of this novel interactive way between humans and robots. To this end, we propose a method to generate a potential grasp configuration relevant to the sketch-depicted objects. Due to the inherent ambiguity of sketches with abstract details, we take the advantage of the graph by incorporating the structure of the sketch to enhance the representation ability. This graph-represented sketch is further validated to improve the generalization of the network, capable of learning the sketch-queried grasp detection by using a small collection (around 100 samples) of hand-drawn sketches. Additionally, our model is trained and tested in an end-to-end manner which is easy to be implemented in real-world applications. Experiments on the multi-object VMRD and GraspNet-1Billion datasets demonstrate the good generalization of the proposed method. The physical robot experiments confirm the utility of our method in object-cluttered scenes.