A High-Fidelity Robotic Manipulator Teleoperation Framework for Human-Centered Augmented Reality Evaluation

TL;DR

This paper introduces ARBot, a real-time teleoperation system using robotic manipulators to accurately record and replay human motions for precise AR validation, addressing variability in human movement.

Contribution

The work presents ARBot, a novel high-fidelity robotic teleoperation platform with new capture models and a safe control algorithm, along with a benchmark dataset for AR evaluation.

Findings

ARBot effectively captures and replays human motions with high fidelity.

The platform enables precise, repeatable ground-truth motion for AR validation.

Open-source dataset supports scalable AR evaluation.

Abstract

Validating Augmented Reality (AR) tracking and interaction models requires precise, repeatable ground-truth motion. However, human users cannot reliably perform consistent motion due to biomechanical variability. Robotic manipulators are promising to act as human motion proxies if they can mimic human movements. In this work, we design and implement ARBot, a real-time teleoperation platform that can effectively capture natural human motion and accurately replay the movements via robotic manipulators. ARBot includes two capture models: stable wrist motion capture via a custom CV and IMU pipeline, and natural 6-DOF control via a mobile application. We design a proactively-safe QP controller to ensure smooth, jitter-free execution of the robotic manipulator, enabling it to function as a high-fidelity record and replay physical proxy. We open-source ARBot and release a benchmark dataset of 132 human and synthetic trajectories captured using ARBot to support controllable and scalable AR evaluation.