Methodical Advice Collection and Reuse in Deep Reinforcement Learning

TL;DR

This paper proposes a method to improve sample efficiency in deep reinforcement learning by using dual uncertainty measures for advice collection and reuse, demonstrated on Atari games.

Contribution

It introduces a novel approach to leverage uncertainties in a teacher-student framework and a new neural network-based method to compute these uncertainties in deep RL.

Findings

Dual uncertainties improve advice efficiency and learning performance.

The method enhances sample efficiency in Atari game benchmarks.

Uncertainty-driven advice collection outperforms baseline approaches.

Abstract

Reinforcement learning (RL) has shown great success in solving many challenging tasks via use of deep neural networks. Although using deep learning for RL brings immense representational power, it also causes a well-known sample-inefficiency problem. This means that the algorithms are data-hungry and require millions of training samples to converge to an adequate policy. One way to combat this issue is to use action advising in a teacher-student framework, where a knowledgeable teacher provides action advice to help the student. This work considers how to better leverage uncertainties about when a student should ask for advice and if the student can model the teacher to ask for less advice. The student could decide to ask for advice when it is uncertain or when both it and its model of the teacher are uncertain. In addition to this investigation, this paper introduces a new method to compute uncertainty for a deep RL agent using a secondary neural network. Our empirical results show that using dual uncertainties to drive advice collection and reuse may improve learning performance across several Atari games.