Neural Map: Structured Memory for Deep Reinforcement Learning

TL;DR

This paper introduces the Neural Map, a structured 2D memory architecture for deep reinforcement learning that enables agents to remember and reason over long-term spatial information in complex environments.

Contribution

The Neural Map provides a novel spatially structured memory system with an adaptable write operator, improving long-term memory capabilities in DRL agents for 3D environments.

Findings

Outperforms previous memory architectures on maze tasks

Capable of generalizing to unseen environments

Effectively stores long-term spatial information

Abstract

A critical component to enabling intelligent reasoning in partially observable environments is memory. Despite this importance, Deep Reinforcement Learning (DRL) agents have so far used relatively simple memory architectures, with the main methods to overcome partial observability being either a temporal convolution over the past k frames or an LSTM layer. More recent work (Oh et al., 2016) has went beyond these architectures by using memory networks which can allow more sophisticated addressing schemes over the past k frames. But even these architectures are unsatisfactory due to the reason that they are limited to only remembering information from the last k frames. In this paper, we develop a memory system with an adaptable write operator that is customized to the sorts of 3D environments that DRL agents typically interact with. This architecture, called the Neural Map, uses a spatially structured 2D memory image to learn to store arbitrary information about the environment over long time lags. We demonstrate empirically that the Neural Map surpasses previous DRL memories on a set of challenging 2D and 3D maze environments and show that it is capable of generalizing to environments that were not seen during training.