Probabilistic design of optimal sequential decision-making algorithms in learning and control

TL;DR

This survey reviews probabilistic sequential decision-making problems in learning and control, emphasizing an optimization framework over probability functions and analyzing how popular algorithms emerge from this perspective.

Contribution

It introduces a unified framework combining problem formulation and resolution methods for probabilistic decision-making, connecting existing algorithms to this approach.

Findings

Revisits popular algorithms through the proposed framework

Highlights the role of infinite-dimensional optimization in decision-making

Provides an open-source example to illustrate the concepts

Abstract

This survey is focused on certain sequential decision-making problems that involve optimizing over probability functions. We discuss the relevance of these problems for learning and control. The survey is organized around a framework that combines a problem formulation and a set of resolution methods. The formulation consists of an infinite-dimensional optimization problem. The methods come from approaches to search optimal solutions in the space of probability functions. Through the lenses of this overarching framework we revisit popular learning and control algorithms, showing that these naturally arise from suitable variations on the formulation mixed with different resolution methods. A running example, for which we make the code available, complements the survey. Finally, a number of challenges arising from the survey are also outlined.