Puppeteer Your Robot: Augmented Reality Leader-Follower Teleoperation

TL;DR

This paper presents an augmented reality system for robot teleoperation that enables intuitive control through virtual puppeteering, reducing physical setup costs while providing enhanced user information.

Contribution

It introduces a novel AR-based teleoperation method that combines physical intuition with virtual control, validated through user studies on manipulation tasks.

Findings

Majority of users rated the system favorably

System effectively enables complex manipulation tasks

Reduces need for physical hardware setup

Abstract

High-quality demonstrations are necessary when learning complex and challenging manipulation tasks. In this work, we introduce an approach to puppeteer a robot by controlling a virtual robot in an augmented reality setting. Our system allows for retaining the advantages of being intuitive from a physical leader-follower side while avoiding the unnecessary use of expensive physical setup. In addition, the user is endowed with additional information using augmented reality. We validate our system with a pilot study n=10 on a block stacking and rice scooping tasks where the majority rates the system favorably. Oculus App and corresponding ROS code are available on the project website: https://ar-puppeteer.github.io/