Rollout Algorithms and Approximate Dynamic Programming for Bayesian Optimization and Sequential Estimation

TL;DR

This paper introduces a unified approximate dynamic programming framework for Bayesian optimization and sequential estimation, utilizing rollout algorithms to improve decision-making in stochastic and deterministic systems.

Contribution

It develops a general framework applying rollout algorithms to Bayesian optimization and sequential estimation, including new approaches for deterministic and stochastic control problems.

Findings

Effective rollout algorithms for Bayesian optimization.

Application of rollout to sequential decoding problems.

Demonstrated improvements in stochastic and deterministic systems.

Abstract

We provide a unifying approximate dynamic programming framework that applies to a broad variety of problems involving sequential estimation. We consider first the construction of surrogate cost functions for the purposes of optimization, and we focus on the special case of Bayesian optimization, using the rollout algorithm and some of its variations. We then discuss the more general case of sequential estimation of a random vector using optimal measurement selection, and its application to problems of stochastic and adaptive control. We distinguish between adaptive control of deterministic and stochastic systems: the former are better suited for the use of rollout, while the latter are well suited for the use of rollout with certainty equivalence approximations. As an example of the deterministic case, we discuss sequential decoding problems, and a rollout algorithm for the approximate solution of the Wordle and Mastermind puzzles, recently developed in the paper [BBB22].