Accelerated Online Reinforcement Learning using Auxiliary Start State Distributions

TL;DR

This paper introduces a method to improve online reinforcement learning by leveraging expert demonstrations and arbitrary reset capabilities, using auxiliary start state distributions to significantly enhance sample efficiency and achieve state-of-the-art results.

Contribution

It proposes a novel approach that uses auxiliary start state distributions informed by safety and episode length to accelerate online RL learning.

Findings

Significant improvement in sample efficiency with auxiliary start state distributions.

State-of-the-art performance on a sparse-reward hard-exploration environment.

Using safety-informed start states accelerates learning process.

Abstract

A long-standing problem in online reinforcement learning (RL) is of ensuring sample efficiency, which stems from an inability to explore environments efficiently. Most attempts at efficient exploration tackle this problem in a setting where learning begins from scratch, without prior information available to bootstrap learning. However, such approaches fail to leverage expert demonstrations and simulators that can reset to arbitrary states. These affordances are valuable resources that offer enormous potential to guide exploration and speed up learning. In this paper, we explore how a small number of expert demonstrations and a simulator allowing arbitrary resets can accelerate learning during online RL. We find that training with a suitable choice of an auxiliary start state distribution that may differ from the true start state distribution of the underlying Markov Decision Process can significantly improve sample efficiency. We find that using a notion of safety to inform the choice of this auxiliary distribution significantly accelerates learning. By using episode length information as a way to operationalize this notion, we demonstrate state-of-the-art sample efficiency on a sparse-reward hard-exploration environment.