Episodic Bayesian Optimal Control with Unknown Randomness Distributions

TL;DR

This paper introduces an episodic Bayesian control method that learns unknown randomness distributions over episodes, converges to optimal policies, and is computationally efficient for convex problems, verified through inventory control experiments.

Contribution

The paper develops a novel episodic Bayesian control framework with convergence guarantees and an efficient SDDP-based computational method for convex stochastic control problems.

Findings

Convergence of policies to true optimal with Bayesian learning.

Asymptotic rate of $O(N^{-1/2})$ for value functions.

Effective numerical performance on inventory control.

Abstract

Stochastic optimal control with unknown randomness distributions has been studied for a long time, encompassing robust control, distributionally robust control, and adaptive control. We propose a new episodic Bayesian approach that incorporates Bayesian learning with optimal control. In each episode, the approach learns the randomness distribution with a Bayesian posterior and subsequently solves the corresponding Bayesian average estimate of the true problem. The resulting policy is exercised during the episode, while additional data/observations of the randomness are collected to update the Bayesian posterior for the next episode. We show that the resulting episodic value functions and policies converge almost surely to their optimal counterparts of the true problem if the parametrized model of the randomness distribution is correctly specified. We further show that the asymptotic convergence rate of the episodic value functions is of the order $O(N^{-1/2})$, where $N$ is the number of episodes given that only one data point is collected in each episode. We develop an efficient computational method based on stochastic dual dynamic programming (SDDP) for a class of problems that have convex cost functions and linear state dynamics. Our numerical results on a classical inventory control problem verify the theoretical convergence results, and numerical comparison with two other methods demonstrate the effectiveness of the proposed Bayesian approach.