VLAD-Grasp: Zero-shot Grasp Detection via Vision-Language Models

TL;DR

VLAD-Grasp leverages vision-language models for zero-shot robotic grasp detection, eliminating the need for curated datasets and enabling generalization to real-world objects with competitive performance.

Contribution

The paper introduces a training-free, zero-shot grasp detection method using vision-language models, advancing robotic manipulation without dataset curation.

Findings

Achieves competitive accuracy on Cornell and Jacquard datasets.

Demonstrates successful zero-shot grasping on real-world objects.

Eliminates the need for curated grasp datasets.

Abstract

Robotic grasping is a fundamental capability for enabling autonomous manipulation, with usually infinite solutions. State-of-the-art approaches for grasping rely on learning from large-scale datasets comprising expert annotations of feasible grasps. Curating such datasets is challenging, and hence, learning-based methods are limited by the solution coverage of the dataset, and require retraining to handle novel objects. Towards this, we present VLAD-Grasp, a Vision-Language model Assisted zero-shot approach for Detecting Grasps. Our method (1) prompts a large vision-language model to generate a goal image where a virtual cylindrical proxy intersects the object's geometry, explicitly encoding an antipodal grasp axis in image space, then (2) predicts depth and segmentation to lift this generated image into 3D, and (3) aligns generated and observed object point clouds via principal components and correspondence-free optimization to recover an executable grasp pose. Unlike prior work, our approach is training-free and does not require curated grasp datasets, while achieving performance competitive with the state-of-the-art methods on the Cornell and Jacquard datasets. Furthermore, we demonstrate zero-shot generalization to real-world objects on a Franka Research 3 robot, highlighting vision-language models as powerful priors for robotic manipulation.