Making Efficient Use of Demonstrations to Solve Hard Exploration Problems

TL;DR

This paper presents R2D3, a reinforcement learning agent that effectively utilizes demonstrations to tackle challenging exploration tasks in complex, partially observable environments, outperforming existing methods on a new benchmark suite.

Contribution

Introduction of R2D3, a novel agent that efficiently leverages demonstrations for hard exploration problems in partially observable settings.

Findings

R2D3 solves several tasks where other methods fail.

R2D3 outperforms state-of-the-art algorithms on the new benchmark suite.

Demonstrations significantly improve exploration efficiency.

Abstract

This paper introduces R2D3, an agent that makes efficient use of demonstrations to solve hard exploration problems in partially observable environments with highly variable initial conditions. We also introduce a suite of eight tasks that combine these three properties, and show that R2D3 can solve several of the tasks where other state of the art methods (both with and without demonstrations) fail to see even a single successful trajectory after tens of billions of steps of exploration.