GLOVER: Generalizable Open-Vocabulary Affordance Reasoning for Task-Oriented Grasping

TL;DR

GLOVER introduces a framework that leverages large language models and a new dataset to enable real-time, open-vocabulary affordance reasoning and grasping for robots, significantly improving speed and accuracy in diverse scenes.

Contribution

The paper presents GLOVER, a novel unified framework that combines LLMs and multi-modal fine-tuning for open-vocabulary affordance reasoning and grasping, with a new dataset and a fast grasp planner.

Findings

Achieves 86.0% success in part identification.

Attains 76.3% success in grasping.

Operates approximately 29 times faster in affordance reasoning than previous methods.

Abstract

Inferring affordable (i.e., graspable) parts of arbitrary objects based on human specifications is essential for robots advancing toward open-vocabulary manipulation. Current grasp planners, however, are hindered by limited vision-language comprehension and time-consuming 3D radiance modeling, restricting real-time, open-vocabulary interactions with objects. To address these limitations, we propose GLOVER, a unified Generalizable Open-Vocabulary Affordance Reasoning framework, which fine-tunes the Large Language Models (LLMs) to predict the visual affordance of graspable object parts within RGB feature space. We compile a dataset of over 10,000 images from human-object interactions, annotated with unified visual and linguistic affordance labels, to enable multi-modal fine-tuning. GLOVER inherits world knowledge and common-sense reasoning from LLMs, facilitating more fine-grained object understanding and sophisticated tool-use reasoning. To enable effective real-world deployment, we present Affordance-Aware Grasping Estimation (AGE), a non-parametric grasp planner that aligns the gripper pose with a superquadric surface derived from affordance data. In evaluations across 30 table-top real-world scenes, GLOVER achieves success rates of 86.0% in part identification and 76.3% in grasping, with speeds approximately 29 times faster in affordance reasoning and 40 times faster in grasping pose estimation than the previous state-of-the-art. We also validate the generalization across embodiments, showing effectiveness in humanoid robots with dexterous hands.