"Give Me an Example Like This": Episodic Active Reinforcement Learning from Demonstrations

TL;DR

This paper introduces EARLY, an active learning algorithm for reinforcement learning from demonstrations, which optimizes the timing and content of queries to improve sample efficiency and human teaching experience.

Contribution

EARLY is a novel episodic active learning method that selectively queries demonstrations in a trajectory-based feature space, enhancing learning efficiency and user experience.

Findings

Achieves expert-level performance in navigation tasks with 30% faster convergence.

Outperforms baseline methods in simulated environments.

Reduces human demonstration time while maintaining learning quality.

Abstract

Reinforcement Learning (RL) has achieved great success in sequential decision-making problems, but often at the cost of a large number of agent-environment interactions. To improve sample efficiency, methods like Reinforcement Learning from Expert Demonstrations (RLED) introduce external expert demonstrations to facilitate agent exploration during the learning process. In practice, these demonstrations, which are often collected from human users, are costly and hence often constrained to a limited amount. How to select the best set of human demonstrations that is most beneficial for learning therefore becomes a major concern. This paper presents EARLY (Episodic Active Learning from demonstration querY), an algorithm that enables a learning agent to generate optimized queries of expert demonstrations in a trajectory-based feature space. Based on a trajectory-level estimate of uncertainty in the agent's current policy, EARLY determines the optimized timing and content for feature-based queries. By querying episodic demonstrations as opposed to isolated state-action pairs, EARLY improves the human teaching experience and achieves better learning performance. We validate the effectiveness of our method in three simulated navigation tasks of increasing difficulty. The results show that our method is able to achieve expert-level performance for all three tasks with convergence over 30\% faster than other baseline methods when demonstrations are generated by simulated oracle policies. The results of a follow-up pilot user study (N=18) further validate that our method can still maintain a significantly better convergence in the case of human expert demonstrators while achieving a better user experience in perceived task load and consuming significantly less human time.