Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulation

TL;DR

This paper introduces Variational State Tabulation (VaST), a method that converts high-dimensional states into a tabular model to improve sample efficiency in deep reinforcement learning, enabling rapid learning and adaptation.

Contribution

The paper proposes VaST, a novel approach that combines variational methods with state tabulation to enhance planning efficiency in high-dimensional environments.

Findings

VaST enables rapid reward maximization in 3D navigation tasks.

It efficiently adapts to sudden changes in rewards or transition dynamics.

The method improves sample efficiency over traditional deep RL algorithms.

Abstract

Modern reinforcement learning algorithms reach super-human performance on many board and video games, but they are sample inefficient, i.e. they typically require significantly more playing experience than humans to reach an equal performance level. To improve sample efficiency, an agent may build a model of the environment and use planning methods to update its policy. In this article we introduce Variational State Tabulation (VaST), which maps an environment with a high-dimensional state space (e.g. the space of visual inputs) to an abstract tabular model. Prioritized sweeping with small backups, a highly efficient planning method, can then be used to update state-action values. We show how VaST can rapidly learn to maximize reward in tasks like 3D navigation and efficiently adapt to sudden changes in rewards or transition probabilities.