Generalizable task-oriented object grasping through LLM-guided ontology and similarity-based planning

TL;DR

This paper presents a geometry-centric, LLM-guided approach to task-oriented object grasping that improves generalization across diverse objects and tasks by using an ontology and similarity-based planning.

Contribution

It introduces a novel ontology and geometric analysis framework that enhances the robustness and generalization of task-oriented grasping without relying on semantic visual features.

Findings

High accuracy in functional part selection and grasp generation.

Effective generalization to novel objects and tasks.

Validation through real-world experiments.

Abstract

Task-oriented grasping (TOG) is more challenging than simple object grasping because it requires precise identification of object parts and careful selection of grasping areas to ensure effective and robust manipulation. While recent approaches have trained large-scale vision-language models to integrate part-level object segmentation with task-aware grasp planning, their instability in part recognition and grasp inference limits their ability to generalize across diverse objects and tasks. To address this issue, we introduce a novel, geometry-centric strategy for more generalizable TOG that does not rely on semantic features from visual recognition, effectively overcoming the viewpoint sensitivity of model-based approaches. Our main proposals include: 1) an object-part-task ontology for functional part selection based on intuitive human commands, constructed using a Large Language Model (LLM); 2) a sampling-based geometric analysis method for identifying the selected object part from observed point clouds, incorporating multiple point distribution and distance metrics; and 3) a similarity matching framework for imitative grasp planning, utilizing similar known objects with pre-existing segmentation and grasping knowledge as references to guide the planning for unknown targets. We validate the high accuracy of our approach in functional part selection, identification, and grasp generation through real-world experiments. Additionally, we demonstrate the method's generalization capabilities to novel-category objects by extending existing ontological knowledge, showcasing its adaptability to a broad range of objects and tasks.