Task-Aware Robotic Grasping by evaluating Quality Diversity Solutions through Foundation Models

TL;DR

This paper introduces a framework combining Large Language Models and Quality Diversity algorithms to enable zero-shot, task-aware robotic grasping by understanding object parts and selecting suitable grasps, validated on real robot experiments.

Contribution

It presents a novel integration of semantic segmentation, LLM prompting, and QD algorithms for task-conditioned grasp synthesis in robotics.

Findings

Achieved 73.6% IoU in predicting task-specific grasp regions.

88% of participants preferred task-aware grasps in validation.

Significant preference for task-aware grasps confirmed by statistical testing.

Abstract

Task-aware robotic grasping is a challenging problem that requires the integration of semantic understanding and geometric reasoning. This paper proposes a novel framework that leverages Large Language Models (LLMs) and Quality Diversity (QD) algorithms to enable zero-shot task-conditioned grasp synthesis. The framework segments objects into meaningful subparts and labels each subpart semantically, creating structured representations that can be used to prompt an LLM. By coupling semantic and geometric representations of an object's structure, the LLM's knowledge about tasks and which parts to grasp can be applied in the physical world. The QD-generated grasp archive provides a diverse set of grasps, allowing us to select the most suitable grasp based on the task. We evaluated the proposed method on a subset of the YCB dataset with a Franka Emika robot. A consolidated ground truth for task-specific grasp regions is established through a survey. Our work achieves a weighted intersection over union (IoU) of 73.6% in predicting task-conditioned grasp regions in 65 task-object combinations. An end-to-end validation study on a smaller subset further confirms the effectiveness of our approach, with 88% of responses favoring the task-aware grasp over the control group. A binomial test shows that participants significantly prefer the task-aware grasp.