Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Control

TL;DR

This paper analyzes how AlphaZero's online decision-making techniques enhance various control strategies, demonstrating broad applicability across deterministic and stochastic problems with continuous and discrete spaces.

Contribution

It provides a unifying mathematical framework showing the generalization of AlphaZero's ideas to diverse optimal and adaptive control problems.

Findings

AlphaZero's rollout and value approximation methods are broadly applicable.

Integration of AlphaZero ideas improves control strategies in various settings.

The framework unifies multiple control and optimization methodologies.

Abstract

In this paper we aim to provide analysis and insights (often based on visualization), which explain the beneficial effects of on-line decision making on top of off-line training. In particular, through a unifying abstract mathematical framework, we show that the principal AlphaZero/TD-Gammon ideas of approximation in value space and rollout apply very broadly to deterministic and stochastic optimal control problems, involving both discrete and continuous search spaces. Moreover, these ideas can be effectively integrated with other important methodologies such as model predictive control, adaptive control, decentralized control, discrete and Bayesian optimization, neural network-based value and policy approximations, and heuristic algorithms for discrete optimization.