Expert-augmented actor-critic for ViZDoom and Montezumas Revenge

TL;DR

This paper introduces an expert-augmented actor-critic algorithm evaluated on sparse reward environments, demonstrating significant performance improvements and surpassing expert data in Montezumas Revenge and ViZDoom.

Contribution

The paper presents a novel expert-augmented actor-critic method that outperforms existing approaches and even surpasses expert data in challenging sparse reward environments.

Findings

Agent scores above 27,000 points on Montezumas Revenge.

Algorithm surpasses the performance of the expert data with proper hyperparameters.

Discovered an unreported bug allowing scores over 800,000 points in Montezumas Revenge.

Abstract

We propose an expert-augmented actor-critic algorithm, which we evaluate on two environments with sparse rewards: Montezumas Revenge and a demanding maze from the ViZDoom suite. In the case of Montezumas Revenge, an agent trained with our method achieves very good results consistently scoring above 27,000 points (in many experiments beating the first world). With an appropriate choice of hyperparameters, our algorithm surpasses the performance of the expert data. In a number of experiments, we have observed an unreported bug in Montezumas Revenge which allowed the agent to score more than 800,000 points.