IFG: Internet-Scale Guidance for Functional Grasping Generation

TL;DR

This paper introduces IFG, a method that combines large vision models with a simulation-based grasping pipeline to enable real-time, precise robotic grasping using internet-scale data and geometric understanding.

Contribution

We propose a novel approach that distills simulation-based grasping data into a diffusion model, enabling real-time, precise 3D grasping without manual data collection.

Findings

Achieves high-performance semantic grasping in cluttered scenes

Operates in real-time on camera point clouds

Does not require manually collected training data

Abstract

Large Vision Models trained on internet-scale data have demonstrated strong capabilities in segmenting and semantically understanding object parts, even in cluttered, crowded scenes. However, while these models can direct a robot toward the general region of an object, they lack the geometric understanding required to precisely control dexterous robotic hands for 3D grasping. To overcome this, our key insight is to leverage simulation with a force-closure grasping generation pipeline that understands local geometries of the hand and object in the scene. Because this pipeline is slow and requires ground-truth observations, the resulting data is distilled into a diffusion model that operates in real-time on camera point clouds. By combining the global semantic understanding of internet-scale models with the geometric precision of a simulation-based locally-aware force-closure, \our achieves high-performance semantic grasping without any manually collected training data. For visualizations of this please visit our website at https://ifgrasping.github.io/