ORACLE-Grasp: Zero-Shot Affordance-Aligned Robotic Grasping using Large Multimodal Models

TL;DR

ORACLE-Grasp introduces a zero-shot, multimodal model-based framework for robotic grasping that combines semantic understanding and spatial reasoning to generalize across diverse objects without task-specific training.

Contribution

The paper presents ORACLE-Grasp, a novel zero-shot approach using large multimodal models for affordance-aligned grasping without training on specific datasets.

Findings

Achieves high success rates in real-world grasping tasks.

Produces human-like, context-sensitive grasp suggestions.

Operates effectively on diverse RGB and RGB-D images.

Abstract

Grasping unknown objects in unstructured environments is a critical challenge for service robots, which must operate in dynamic, real-world settings such as homes, hospitals, and warehouses. Success in these environments requires both semantic understanding and spatial reasoning. Traditional methods often rely on dense training datasets or detailed geometric modeling, which demand extensive data collection and do not generalize well to novel objects or affordances. We present ORACLE-Grasp, a zero-shot framework that leverages Large Multimodal Models (LMMs) as semantic oracles to guide affordance-aligned grasp selection, without requiring task-specific training or manual input. The system reformulates grasp prediction as a structured, iterative decision process, using a dual-prompt tool-calling strategy: the first prompt extracts high-level object semantics, while the second identifies graspable regions aligned with the object's function. To address the spatial limitations of LMMs, ORACLE-Grasp discretizes the image into candidate regions and reasons over them to produce human-like and context-sensitive grasp suggestions. A depth-based refinement step improves grasp reliability when available, and an early stopping mechanism enhances computational efficiency. We evaluate ORACLE-Grasp on a diverse set of RGB and RGB-D images featuring both everyday and AI-generated objects. The results show that our method produces physically feasible and semantically appropriate grasps that align closely with human annotations, achieving high success rates in real-world pick-up tasks. Our findings highlight the potential of LMMs for enabling flexible and generalizable grasping strategies in autonomous service robots, eliminating the need for object-specific models or extensive training.