Active Hierarchical Imitation and Reinforcement Learning

TL;DR

This paper introduces an active hierarchical imitation and reinforcement learning framework that improves learning efficiency and reduces human effort in training robots to solve complex tasks, demonstrated through maze navigation experiments.

Contribution

It develops active learning algorithms within hierarchical imitation and reinforcement learning, enhancing efficiency and reducing human effort in robot training.

Findings

Active learning methods outperform traditional approaches in efficiency.

DAgger and reward-based active learning improve performance.

Human effort is significantly reduced during training.

Abstract

Humans can leverage hierarchical structures to split a task into sub-tasks and solve problems efficiently. Both imitation and reinforcement learning or a combination of them with hierarchical structures have been proven to be an efficient way for robots to learn complex tasks with sparse rewards. However, in the previous work of hierarchical imitation and reinforcement learning, the tested environments are in relatively simple 2D games, and the action spaces are discrete. Furthermore, many imitation learning works focusing on improving the policies learned from the expert polices that are hard-coded or trained by reinforcement learning algorithms, rather than human experts. In the scenarios of human-robot interaction, humans can be required to provide demonstrations to teach the robot, so it is crucial to improve the learning efficiency to reduce expert efforts, and know human's perception about the learning/training process. In this project, we explored different imitation learning algorithms and designed active learning algorithms upon the hierarchical imitation and reinforcement learning framework we have developed. We performed an experiment where five participants were asked to guide a randomly initialized agent to a random goal in a maze. Our experimental results showed that using DAgger and reward-based active learning method can achieve better performance while saving more human efforts physically and mentally during the training process.