The Predictron: End-To-End Learning and Planning

TL;DR

The paper introduces the predictron, an end-to-end trainable architecture that models planning as a Markov reward process, enabling more accurate value predictions in complex environments.

Contribution

It presents the predictron architecture, a novel fully abstract model that performs multi-step imagined planning within a neural network framework.

Findings

Outperforms conventional neural networks in maze and pool simulations

Accurately approximates true value functions through multi-step internal predictions

Demonstrates effectiveness in procedurally generated environments

Abstract

One of the key challenges of artificial intelligence is to learn models that are effective in the context of planning. In this document we introduce the predictron architecture. The predictron consists of a fully abstract model, represented by a Markov reward process, that can be rolled forward multiple "imagined" planning steps. Each forward pass of the predictron accumulates internal rewards and values over multiple planning depths. The predictron is trained end-to-end so as to make these accumulated values accurately approximate the true value function. We applied the predictron to procedurally generated random mazes and a simulator for the game of pool. The predictron yielded significantly more accurate predictions than conventional deep neural network architectures.