XLVIN: eXecuted Latent Value Iteration Nets

TL;DR

XLVINs extend Value Iteration Networks by integrating contrastive learning, graph reasoning, and neural algorithms, enabling planning in more general environments and improving performance over model-free methods.

Contribution

The paper introduces XLVINs, a novel approach that overcomes VIN limitations by combining recent AI techniques for broader applicability and enhanced performance.

Findings

XLVINs match VIN performance on discrete, fixed MDPs.

XLVINs outperform model-free baselines in various environments.

The approach generalizes planning to more complex, unknown environments.

Abstract

Value Iteration Networks (VINs) have emerged as a popular method to incorporate planning algorithms within deep reinforcement learning, enabling performance improvements on tasks requiring long-range reasoning and understanding of environment dynamics. This came with several limitations, however: the model is not incentivised in any way to perform meaningful planning computations, the underlying state space is assumed to be discrete, and the Markov decision process (MDP) is assumed fixed and known. We propose eXecuted Latent Value Iteration Networks (XLVINs), which combine recent developments across contrastive self-supervised learning, graph representation learning and neural algorithmic reasoning to alleviate all of the above limitations, successfully deploying VIN-style models on generic environments. XLVINs match the performance of VIN-like models when the underlying MDP is discrete, fixed and known, and provides significant improvements to model-free baselines across three general MDP setups.