SAM2Grasp: Resolve Multi-modal Grasping via Prompt-conditioned Temporal Action Prediction

TL;DR

SAM2Grasp introduces a prompt-conditioned, temporal action prediction framework that resolves multimodal grasping ambiguity, enabling continuous, unambiguous robotic grasping in cluttered scenes by leveraging a frozen visual model and a lightweight trainable head.

Contribution

The paper presents a novel approach that reformulates multi-object grasping as a prompt-conditioned prediction task, utilizing a frozen visual model with a trainable action head for improved performance.

Findings

Achieves state-of-the-art results in multi-object grasping tasks.

Effectively eliminates ambiguity in visuomotor policies.

Maintains stable object tracking during grasping in cluttered scenes.

Abstract

Imitation learning for robotic grasping is often plagued by the multimodal problem: when a scene contains multiple valid targets, demonstrations of grasping different objects create conflicting training signals. Standard imitation learning policies fail by averaging these distinct actions into a single, invalid action. In this paper, we introduce SAM2Grasp, a novel framework that resolves this issue by reformulating the task as a uni-modal, prompt-conditioned prediction problem. Our method leverages the frozen SAM2 model to use its powerful visual temporal tracking capability and introduces a lightweight, trainable action head that operates in parallel with its native segmentation head. This design allows for training only the small action head on pre-computed temporal-visual features from SAM2. During inference, an initial prompt, such as a bounding box provided by an upstream object detection model, designates the specific object to be grasped. This prompt conditions the action head to predict a unique, unambiguous grasp trajectory for that object alone. In all subsequent video frames, SAM2's built-in temporal tracking capability automatically maintains stable tracking of the selected object, enabling our model to continuously predict the grasp trajectory from the video stream without further external guidance. This temporal-prompted approach effectively eliminates ambiguity from the visuomotor policy. We demonstrate through extensive experiments that SAM2Grasp achieves state-of-the-art performance in cluttered, multi-object grasping tasks.