Learning robot motor skills with mixed reality

TL;DR

This paper presents a mixed reality-based framework for teaching robots complex motor skills by integrating demonstrations, constraints, planning, and object info into a unified learning system using Dynamic Movement Primitives.

Contribution

It introduces a novel MR interface for intuitive teaching of diverse motor skills and a sample-efficient learning framework that combines multiple modalities of world knowledge.

Findings

Effective integration of multimodal inputs into DMPs.

Enhanced teaching efficiency with MR interface.

Potential for robots to learn complex tasks more effectively.

Abstract

Mixed Reality (MR) has recently shown great success as an intuitive interface for enabling end-users to teach robots. Related works have used MR interfaces to communicate robot intents and beliefs to a co-located human, as well as developed algorithms for taking multi-modal human input and learning complex motor behaviors. Even with these successes, enabling end-users to teach robots complex motor tasks still poses a challenge because end-user communication is highly task dependent and world knowledge is highly varied. We propose a learning framework where end-users teach robots a) motion demonstrations, b) task constraints, c) planning representations, and d) object information, all of which are integrated into a single motor skill learning framework based on Dynamic Movement Primitives (DMPs). We hypothesize that conveying this world knowledge will be intuitive with an MR interface, and that a sample-efficient motor skill learning framework which incorporates varied modalities of world knowledge will enable robots to effectively solve complex tasks.