Blending Imitation and Reinforcement Learning for Robust Policy Improvement

TL;DR

This paper introduces RPI, a novel algorithm that combines imitation and reinforcement learning to improve sample efficiency and robustness in policy learning, especially in sparse-reward environments.

Contribution

It presents a new method that interleaves IL and RL, using online performance estimates to adaptively switch between them, outperforming existing approaches.

Findings

RPI achieves superior performance on benchmark tasks.

Theoretical analysis confirms its robustness and efficiency.

Empirical results show effective learning from diverse oracles.

Abstract

While reinforcement learning (RL) has shown promising performance, its sample complexity continues to be a substantial hurdle, restricting its broader application across a variety of domains. Imitation learning (IL) utilizes oracles to improve sample efficiency, yet it is often constrained by the quality of the oracles deployed. which actively interleaves between IL and RL based on an online estimate of their performance. RPI draws on the strengths of IL, using oracle queries to facilitate exploration, an aspect that is notably challenging in sparse-reward RL, particularly during the early stages of learning. As learning unfolds, RPI gradually transitions to RL, effectively treating the learned policy as an improved oracle. This algorithm is capable of learning from and improving upon a diverse set of black-box oracles. Integral to RPI are Robust Active Policy Selection (RAPS) and Robust Policy Gradient (RPG), both of which reason over whether to perform state-wise imitation from the oracles or learn from its own value function when the learner's performance surpasses that of the oracles in a specific state. Empirical evaluations and theoretical analysis validate that RPI excels in comparison to existing state-of-the-art methodologies, demonstrating superior performance across various benchmark domains.