MirrorLimb: Implementing hand pose acquisition and robot teleoperation based on RealMirror

TL;DR

MirrorLimb introduces a cost-effective, real-time hand pose acquisition and robot teleoperation framework compatible with RealMirror, enhancing robotic manipulation research and dataset creation.

Contribution

It provides a novel low-cost, real-time hand pose tracking and teleoperation system integrated with RealMirror for robotic manipulation and dataset development.

Findings

Outperforms mainstream visual tracking in cost and accuracy.

Enables real-time teleoperation of various robotic end-effectors.

Facilitates construction of Vision-Language-Action datasets.

Abstract

In this work, we present a PICO-based robot remote operating framework that enables low-cost, real-time acquisition of hand motion and pose data, outperforming mainstream visual tracking and motion capture solutions in terms of cost-effectiveness. The framework is natively compatible with the RealMirror ecosystem, offering ready-to-use functionality for stable and precise robotic trajectory recording within the Isaac simulation environment, thereby facilitating the construction of Vision-Language-Action (VLA) datasets. Additionally, the system supports real-time teleoperation of a variety of end-effector-equipped robots, including dexterous hands and robotic grippers. This work aims to lower the technical barriers in the study of upper-limb robotic manipulation, thereby accelerating advancements in VLA-related research.