Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

TL;DR

This paper presents a method using virtual reality teleoperation with consumer hardware to collect demonstrations for training deep neural networks in complex robot manipulation tasks, enabling robots to learn visuomotor skills.

Contribution

It introduces a practical approach combining VR teleoperation and imitation learning to acquire complex robot manipulation skills from raw visual data.

Findings

VR teleoperation enables natural demonstration collection

Deep neural networks successfully learn visuomotor policies

Effective for complex manipulation tasks

Abstract

Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor skills.