Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environments

TL;DR

This paper introduces imitator learning, a method enabling agents to quickly adapt to new tasks and environments using minimal demonstrations, by integrating imitation learning with reinforcement learning and attention mechanisms.

Contribution

The paper proposes a novel imitator learning framework with Demo-Attention Actor-Critic that adapts policies on-the-fly from limited demonstrations without extra tuning.

Findings

DAAC outperforms previous imitation methods significantly

Effective in unseen tasks and environments

Developed new benchmarks for navigation and robotics

Abstract

Imitation learning (IL) enables agents to mimic expert behaviors. Most previous IL techniques focus on precisely imitating one policy through mass demonstrations. However, in many applications, what humans require is the ability to perform various tasks directly through a few demonstrations of corresponding tasks, where the agent would meet many unexpected changes when deployed. In this scenario, the agent is expected to not only imitate the demonstration but also adapt to unforeseen environmental changes. This motivates us to propose a new topic called imitator learning (ItorL), which aims to derive an imitator module that can on-the-fly reconstruct the imitation policies based on very limited expert demonstrations for different unseen tasks, without any extra adjustment. In this work, we focus on imitator learning based on only one expert demonstration. To solve ItorL, we propose Demo-Attention Actor-Critic (DAAC), which integrates IL into a reinforcement-learning paradigm that can regularize policies' behaviors in unexpected situations. Besides, for autonomous imitation policy building, we design a demonstration-based attention architecture for imitator policy that can effectively output imitated actions by adaptively tracing the suitable states in demonstrations. We develop a new navigation benchmark and a robot environment for \topic~and show that DAAC~outperforms previous imitation methods \textit{with large margins} both on seen and unseen tasks.