Many-Goals Reinforcement Learning

TL;DR

This paper investigates the application of many-goals reinforcement learning in deep neural network settings, exploring its use for mastery, pre-training, and auxiliary tasks to improve learning efficiency and performance.

Contribution

It extends the concept of many-goals updating from small-state tabular RL to deep RL, demonstrating its effectiveness in mastery, pre-training, and auxiliary learning tasks.

Findings

Many-goals updating improves learning in visual domains.

Pre-training with many-goals accelerates task learning.

Auxiliary goals enhance main task performance.

Abstract

All-goals updating exploits the off-policy nature of Q-learning to update all possible goals an agent could have from each transition in the world, and was introduced into Reinforcement Learning (RL) by Kaelbling (1993). In prior work this was mostly explored in small-state RL problems that allowed tabular representations and where all possible goals could be explicitly enumerated and learned separately. In this paper we empirically explore 3 different extensions of the idea of updating many (instead of all) goals in the context of RL with deep neural networks (or DeepRL for short). First, in a direct adaptation of Kaelbling's approach we explore if many-goals updating can be used to achieve mastery in non-tabular visual-observation domains. Second, we explore whether many-goals updating can be used to pre-train a network to subsequently learn faster and better on a single main task of interest. Third, we explore whether many-goals updating can be used to provide auxiliary task updates in training a network to learn faster and better on a single main task of interest. We provide comparisons to baselines for each of the 3 extensions.