Language-Driven Closed-Loop Grasping with Model-Predictive Trajectory Replanning

TL;DR

This paper introduces a modular, zero-shot framework for language-guided robotic manipulation that enables real-time tracking and trajectory replanning to grasp moving objects smoothly in dynamic environments.

Contribution

The work presents a novel modular framework integrating vision-language models with real-time pose estimation and trajectory planning for dynamic object grasping.

Findings

Achieves up to 30 Hz update rate for pose localization.

Operates at 10 Hz for trajectory optimization.

Successfully grasps moving objects in real-time.

Abstract

Combining a vision module inside a closed-loop control system for a \emph{seamless movement} of a robot in a manipulation task is challenging due to the inconsistent update rates between utilized modules. This task is even more difficult in a dynamic environment, e.g., objects are moving. This paper presents a \emph{modular} zero-shot framework for language-driven manipulation of (dynamic) objects through a closed-loop control system with real-time trajectory replanning and an online 6D object pose localization. We segment an object within $\SI{0.5}{\second}$ by leveraging a vision language model via language commands. Then, guided by natural language commands, a closed-loop system, including a unified pose estimation and tracking and online trajectory planning, is utilized to continuously track this object and compute the optimal trajectory in real-time. Our proposed zero-shot framework provides a smooth trajectory that avoids jerky movements and ensures the robot can grasp a non-stationary object. Experiment results exhibit the real-time capability of the proposed zero-shot modular framework for the trajectory optimization module to accurately and efficiently grasp moving objects, i.e., up to \SI{30}{\hertz} update rates for the online 6D pose localization module and \SI{10}{\hertz} update rates for the receding-horizon trajectory optimization. These advantages highlight the modular framework's potential applications in robotics and human-robot interaction; see the video in https://www.acin.tuwien.ac.at/en/6e64/.