Robustness to incorrect priors in partially observed stochastic control

TL;DR

This paper investigates how the optimal solutions to stochastic control problems behave when the initial probability priors are inaccurate, establishing conditions for robustness and continuity under different types of convergence.

Contribution

It demonstrates that optimal cost is continuous under total variation convergence and provides bounds on control mismatch errors due to prior inaccuracies, under certain measurement assumptions.

Findings

Optimal cost is continuous under total variation convergence.

Robustness and continuity can hold under weak convergence with additional assumptions.

Bounds on control mismatch errors due to prior inaccuracies are derived.

Abstract

We study the continuity properties of optimal solutions to stochastic control problems with respect to initial probability measures and applications of these to the robustness of optimal control policies applied to systems with incomplete or incorrect priors. It is shown that for single and multi-stage optimal cost problems, continuity and robustness cannot be established under weak convergence or Wasserstein convergence in general, but that the optimal cost is continuous in the priors under the convergence in total variation under mild conditions. By imposing further assumptions on the measurement models, robustness and continuity also hold under weak convergence of priors. We thus obtain robustness results and bounds on the mismatch error that occurs due to the application of a control policy which is designed for an incorrectly estimated prior in terms of a distance measure between the true prior and the incorrect one. Positive and negative practical implications of these results in empirical learning for stochastic control will be presented, where almost surely weak convergence of i.i.d. empirical measures occurs but stronger notions of convergence, such as total variation convergence, in general, do not.