Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms

TL;DR

This paper introduces a framework for active imitation learning from multiple non-deterministic teachers, focusing on learning a policy distribution efficiently while minimizing interaction costs.

Contribution

It proposes a novel framework for modeling policy distributions and introduces APIL, an active learning algorithm that reduces interaction costs in uncertain teacher scenarios.

Findings

APIL significantly reduces teacher interactions.

APIL maintains performance despite behavioral uncertainty.

Framework effectively models non-deterministic teacher policies.

Abstract

We formulate the problem of learning to imitate multiple, non-deterministic teachers with minimal interaction cost. Rather than learning a specific policy as in standard imitation learning, the goal in this problem is to learn a distribution over a policy space. We first present a general framework that efficiently models and estimates such a distribution by learning continuous representations of the teacher policies. Next, we develop Active Performance-Based Imitation Learning (APIL), an active learning algorithm for reducing the learner-teacher interaction cost in this framework. By making query decisions based on predictions of future progress, our algorithm avoids the pitfalls of traditional uncertainty-based approaches in the face of teacher behavioral uncertainty. Results on both toy and photo-realistic navigation tasks show that APIL significantly reduces the numbers of interactions with teachers without compromising on performance. Moreover, it is robust to various degrees of teacher behavioral uncertainty.