Representation Learning on Graphs: A Reinforcement Learning Application

TL;DR

This paper explores advanced representation learning techniques on graphs to improve value function approximation in high-dimensional reinforcement learning, demonstrating that graph-based features outperform traditional methods.

Contribution

It introduces the use of graph representation learning algorithms like node2vec and Variational Graph Auto-Encoder for RL value function approximation, surpassing proto-value functions.

Findings

Node2Vec and Variational Graph Auto-Encoder outperform proto-value functions in experiments.

Graph-based features improve low-dimensional value function approximation.

Representation learning enhances RL performance in high-dimensional spaces.

Abstract

In this work, we study value function approximation in reinforcement learning (RL) problems with high dimensional state or action spaces via a generalized version of representation policy iteration (RPI). We consider the limitations of proto-value functions (PVFs) at accurately approximating the value function in low dimensions and we highlight the importance of features learning for an improved low-dimensional value function approximation. Then, we adopt different representation learning algorithm on graphs to learn the basis functions that best represent the value function. We empirically show that node2vec, an algorithm for scalable feature learning in networks, and the Variational Graph Auto-Encoder constantly outperform the commonly used smooth proto-value functions in low-dimensional feature space.