Interactive Human-in-the-loop Coordination of Manipulation Skills Learned from Demonstration

TL;DR

This paper introduces a human-in-the-loop framework for learning and coordinating complex robot manipulation skills from demonstrations, enabling adaptable and efficient task execution without relying on precise simulation.

Contribution

It presents a novel framework combining parameterized skill models, geometric task networks, and hierarchical control learned incrementally during execution.

Findings

Successfully teaches industrial tasks in less than 30 minutes

Reduces manual design efforts significantly

Improves adaptability to new scenes

Abstract

Learning from demonstration (LfD) provides a fast, intuitive and efficient framework to program robot skills, which has gained growing interest both in research and industrial applications. Most complex manipulation tasks are long-term and involve a set of skill primitives. Thus it is crucial to have a reliable coordination scheme that selects the correct sequence of skill primitive and the correct parameters for each skill, under various scenarios. Instead of relying on a precise simulator, this work proposes a human-in-the-loop coordination framework for LfD skills that: builds parameterized skill models from kinesthetic demonstrations; constructs a geometric task network (GTN) on-the-fly from human instructions; learns a hierarchical control policy incrementally during execution. This framework can reduce significantly the manual design efforts, while improving the adaptability to new scenes. We show on a 7-DoF robotic manipulator that the proposed approach can teach complex industrial tasks such as bin sorting and assembly in less than 30 minutes.