The Dreaming Variational Autoencoder for Reinforcement Learning Environments

TL;DR

This paper introduces the Dreaming Variational Autoencoder (DVAE), a generative model designed to enhance exploration in reinforcement learning environments, especially those with sparse feedback, using a new maze engine called Deep Maze.

Contribution

The paper proposes DVAE, a novel generative architecture for exploration, and introduces Deep Maze, a flexible environment for testing reinforcement learning algorithms in complex scenarios.

Findings

Initial results show DVAE improves exploration in sparse feedback environments.

Deep Maze effectively challenges RL algorithms in diverse scenarios.

Encourages further research in generative exploration methods.

Abstract

Reinforcement learning has shown great potential in generalizing over raw sensory data using only a single neural network for value optimization. There are several challenges in the current state-of-the-art reinforcement learning algorithms that prevent them from converging towards the global optima. It is likely that the solution to these problems lies in short- and long-term planning, exploration and memory management for reinforcement learning algorithms. Games are often used to benchmark reinforcement learning algorithms as they provide a flexible, reproducible, and easy to control environment. Regardless, few games feature a state-space where results in exploration, memory, and planning are easily perceived. This paper presents The Dreaming Variational Autoencoder (DVAE), a neural network based generative modeling architecture for exploration in environments with sparse feedback. We further present Deep Maze, a novel and flexible maze engine that challenges DVAE in partial and fully-observable state-spaces, long-horizon tasks, and deterministic and stochastic problems. We show initial findings and encourage further work in reinforcement learning driven by generative exploration.