Point2Act: Efficient 3D Distillation of Multimodal LLMs for Zero-Shot Context-Aware Grasping

TL;DR

Point2Act introduces an efficient method for zero-shot, context-aware 3D action localization in robotics by leveraging multimodal large language models and multi-view aggregation, enabling rapid and precise physical actions.

Contribution

It presents a novel 3D relevancy field approach that bypasses high-dimensional features, improving localization accuracy for robotic grasping tasks in unseen environments.

Findings

Achieves scene understanding and action localization in under 20 seconds.

Effectively compensates for occlusion and semantic uncertainties.

Enables precise 3D action point retrieval for manipulation tasks.

Abstract

We propose Point2Act, which directly retrieves the 3D action point relevant to a contextually described task, leveraging Multimodal Large Language Models (MLLMs). Foundation models opened the possibility for generalist robots that can perform a zero-shot task following natural language descriptions within an unseen environment. While the semantics obtained from large-scale image and language datasets provide contextual understanding in 2D images, the rich yet nuanced features deduce blurry 2D regions and struggle to find precise 3D locations for actions. Our proposed 3D relevancy fields bypass the high-dimensional features and instead efficiently imbue lightweight 2D point-level guidance tailored to the task-specific action. The multi-view aggregation effectively compensates for misalignments due to geometric ambiguities, such as occlusion, or semantic uncertainties inherent in the language descriptions. The output region is highly localized, reasoning fine-grained 3D spatial context that can directly transfer to an explicit position for physical action at the on-the-fly reconstruction of the scene. Our full-stack pipeline, which includes capturing, MLLM querying, 3D reconstruction, and grasp pose extraction, generates spatially grounded responses in under 20 seconds, facilitating practical manipulation tasks. Project page: https://sangminkim-99.github.io/point2act/