Jointly Pre-training with Supervised, Autoencoder, and Value Losses for Deep Reinforcement Learning

TL;DR

This paper introduces a joint pre-training strategy combining supervised, autoencoder, and value losses to incorporate human knowledge into Deep Reinforcement Learning, significantly improving learning efficiency and performance in Atari games.

Contribution

It proposes a novel pre-training method that jointly optimizes multiple losses, enhancing feature learning and accelerating DRL training with human demonstrations.

Findings

Pre-training improves Atari game performance with fewer interactions.

The method outperforms state-of-the-art algorithms in Pong and MsPacman.

Pre-training is lightweight and easy to implement.

Abstract

Deep Reinforcement Learning (DRL) algorithms are known to be data inefficient. One reason is that a DRL agent learns both the feature and the policy tabula rasa. Integrating prior knowledge into DRL algorithms is one way to improve learning efficiency since it helps to build helpful representations. In this work, we consider incorporating human knowledge to accelerate the asynchronous advantage actor-critic (A3C) algorithm by pre-training a small amount of non-expert human demonstrations. We leverage the supervised autoencoder framework and propose a novel pre-training strategy that jointly trains a weighted supervised classification loss, an unsupervised reconstruction loss, and an expected return loss. The resulting pre-trained model learns more useful features compared to independently training in supervised or unsupervised fashion. Our pre-training method drastically improved the learning performance of the A3C agent in Atari games of Pong and MsPacman, exceeding the performance of the state-of-the-art algorithms at a much smaller number of game interactions. Our method is light-weight and easy to implement in a single machine. For reproducibility, our code is available at github.com/gabrieledcjr/DeepRL/tree/A3C-ALA2019