Visually Guided Model Predictive Robot Control via 6D Object Pose Localization and Tracking

TL;DR

This paper presents a real-time, accurate visual perception and control system enabling robots to manipulate moving objects by estimating their 6D pose at high frequency and integrating this with model predictive control for stable operation.

Contribution

It introduces a novel asynchronous visual perception module combining learning-based and model-based 6D pose estimation, and integrates it with a torque-based MPC for dynamic object manipulation.

Findings

Achieves 120 Hz 6D pose estimation with high accuracy.

Demonstrates stable robot control of moving objects in real-world experiments.

Validates approach on challenging benchmarks and real robot tasks.

Abstract

The objective of this work is to enable manipulation tasks with respect to the 6D pose of a dynamically moving object using a camera mounted on a robot. Examples include maintaining a constant relative 6D pose of the robot arm with respect to the object, grasping the dynamically moving object, or co-manipulating the object together with a human. Fast and accurate 6D pose estimation is crucial to achieve smooth and stable robot control in such situations. The contributions of this work are three fold. First, we propose a new visual perception module that asynchronously combines accurate learning-based 6D object pose localizer and a high-rate model-based 6D pose tracker. The outcome is a low-latency accurate and temporally consistent 6D object pose estimation from the input video stream at up to 120 Hz. Second, we develop a visually guided robot arm controller that combines the new visual perception module with a torque-based model predictive control algorithm. Asynchronous combination of the visual and robot proprioception signals at their corresponding frequencies results in stable and robust 6D object pose guided robot arm control. Third, we experimentally validate the proposed approach on a challenging 6D pose estimation benchmark and demonstrate 6D object pose-guided control with dynamically moving objects on a real 7 DoF Franka Emika Panda robot.