Autoverse: An Evolvable Game Language for Learning Robust Embodied Agents

TL;DR

Autoverse is a flexible, evolvable game language that accelerates reinforcement learning by generating complex environments and using imitation learning to bootstrap robust embodied agents.

Contribution

The paper introduces Autoverse, a domain-specific language for 2D grid games, enabling scalable environment generation and a novel curriculum-based approach for open-ended learning.

Findings

Accelerates RL training via GPU-parallelized environment simulation.

Evolves environments to challenge search algorithms, creating a curriculum.

Improves agent performance and generality through imitation and environment evolution.

Abstract

We introduce Autoverse, an evolvable, domain-specific language for single-player 2D grid-based games, and demonstrate its use as a scalable training ground for Open-Ended Learning (OEL) algorithms. Autoverse uses cellular-automaton-like rewrite rules to describe game mechanics, allowing it to express various game environments (e.g. mazes, dungeons, sokoban puzzles) that are popular testbeds for Reinforcement Learning (RL) agents. Each rewrite rule can be expressed as a series of simple convolutions, allowing for environments to be parallelized on the GPU, thereby drastically accelerating RL training. Using Autoverse, we propose jump-starting open-ended learning by imitation learning from search. In such an approach, we first evolve Autoverse environments (their rules and initial map topology) to maximize the number of iterations required by greedy tree search to discover a new best solution, producing a curriculum of increasingly complex environments and playtraces. We then distill these expert playtraces into a neural-network-based policy using imitation learning. Finally, we use the learned policy as a starting point for open-ended RL, where new training environments are continually evolved to maximize the RL player agent's value function error (a proxy for its regret, or the learnability of generated environments), finding that this approach improves the performance and generality of resultant player agents.