Graph Convolutional Memory using Topological Priors

TL;DR

This paper introduces graph convolutional memory (GCM), a novel hybrid memory model that leverages topological priors to improve reinforcement learning in partially observable environments, outperforming existing methods especially with human priors.

Contribution

The paper presents GCM, the first hybrid memory model for POMDPs that integrates topological priors with graph convolution, enhancing performance and parameter efficiency.

Findings

GCM performs comparably to state-of-the-art methods without priors.

With human priors, GCM surpasses existing methods on control, memorization, and navigation tasks.

GCM uses fewer parameters while achieving better results.

Abstract

Solving partially-observable Markov decision processes (POMDPs) is critical when applying reinforcement learning to real-world problems, where agents have an incomplete view of the world. We present graph convolutional memory (GCM), the first hybrid memory model for solving POMDPs using reinforcement learning. GCM uses either human-defined or data-driven topological priors to form graph neighborhoods, combining them into a larger network topology using dynamic programming. We query the graph using graph convolution, coalescing relevant memories into a context-dependent belief. When used without human priors, GCM performs similarly to state-of-the-art methods. When used with human priors, GCM outperforms these methods on control, memorization, and navigation tasks while using significantly fewer parameters.