Optimal Agnostic Control of Unknown Linear Dynamics in a Bounded Parameter Range

TL;DR

This paper explores optimal control strategies for unknown linear dynamics with bounded parameters, comparing Bayesian and agnostic approaches, and introduces methods to minimize expected cost and regret under uncertainty.

Contribution

It develops a framework for solving optimal control problems with unknown parameters by reducing agnostic control to Bayesian control using finite priors.

Findings

Optimal strategies derived via Hamilton-Jacobi-Bellman PDEs

Reduction of agnostic control to Bayesian control with finite priors

Strategies minimize expected cost and regret under parameter uncertainty

Abstract

Here and in a follow-on paper, we consider a simple control problem in which the underlying dynamics depend on a parameter $a$ that is unknown and must be learned. In this paper, we assume that $a$ is bounded, i.e., that $|a| \le a_{\text{MAX}}$, and we study two variants of the control problem. In the first variant, Bayesian control, we are given a prior probability distribution for $a$ and we seek a strategy that minimizes the expected value of a given cost function. Assuming that we can solve a certain PDE (the Hamilton-Jacobi-Bellman equation), we produce optimal strategies for Bayesian control. In the second variant, agnostic control, we assume nothing about $a$ and we seek a strategy that minimizes a quantity called the regret. We produce a prior probability distribution $d\text{Prior}(a)$ supported on a finite subset of $[-a_{\text{MAX}},a_{\text{MAX}}]$ so that the agnostic control problem reduces to the Bayesian control problem for the prior $d\text{Prior}(a)$.