Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning

TL;DR

This paper introduces Theory-Based Reinforcement Learning, a model-based approach using human-like causal theories to enable rapid, human-like learning and generalization in complex environments, exemplified by a video game agent.

Contribution

It presents a novel RL framework that incorporates intuitive theories for efficient learning, demonstrated through a versatile agent that learns multiple games quickly and generalizes well.

Findings

EMPA learns new games in minutes

EMPA generalizes to new levels and situations

Model captures human-like exploration and learning dynamics

Abstract

Reinforcement learning (RL) studies how an agent comes to achieve reward in an environment through interactions over time. Recent advances in machine RL have surpassed human expertise at the world's oldest board games and many classic video games, but they require vast quantities of experience to learn successfully -- none of today's algorithms account for the human ability to learn so many different tasks, so quickly. Here we propose a new approach to this challenge based on a particularly strong form of model-based RL which we call Theory-Based Reinforcement Learning, because it uses human-like intuitive theories -- rich, abstract, causal models of physical objects, intentional agents, and their interactions -- to explore and model an environment, and plan effectively to achieve task goals. We instantiate the approach in a video game playing agent called EMPA (the Exploring, Modeling, and Planning Agent), which performs Bayesian inference to learn probabilistic generative models expressed as programs for a game-engine simulator, and runs internal simulations over these models to support efficient object-based, relational exploration and heuristic planning. EMPA closely matches human learning efficiency on a suite of 90 challenging Atari-style video games, learning new games in just minutes of game play and generalizing robustly to new game situations and new levels. The model also captures fine-grained structure in people's exploration trajectories and learning dynamics. Its design and behavior suggest a way forward for building more general human-like AI systems.