Scaling All-Goals Updates in Reinforcement Learning Using Convolutional Neural Networks

TL;DR

This paper introduces a convolutional neural network approach to efficiently learn and generalize all-goals updates in reinforcement learning, enabling scalable navigation and exploration in complex environments.

Contribution

The authors propose a novel convolutional network architecture for all-goals updates, significantly improving scalability and generalization in reinforcement learning tasks.

Findings

Effective in maze and Sokoban environments

Improves exploration in Montezuma's Revenge and Super Mario

Achieves accurate goal distance mapping

Abstract

Being able to reach any desired location in the environment can be a valuable asset for an agent. Learning a policy to navigate between all pairs of states individually is often not feasible. An all-goals updating algorithm uses each transition to learn Q-values towards all goals simultaneously and off-policy. However the expensive numerous updates in parallel limited the approach to small tabular cases so far. To tackle this problem we propose to use convolutional network architectures to generate Q-values and updates for a large number of goals at once. We demonstrate the accuracy and generalization qualities of the proposed method on randomly generated mazes and Sokoban puzzles. In the case of on-screen goal coordinates the resulting mapping from frames to distance-maps directly informs the agent about which places are reachable and in how many steps. As an example of application we show that replacing the random actions in epsilon-greedy exploration by several actions towards feasible goals generates better exploratory trajectories on Montezuma's Revenge and Super Mario All-Stars games.