Constrained Policy Optimization for Stochastic Optimal Control under Nonstationary Uncertainties

TL;DR

This paper develops a constrained policy optimization method for stochastic optimal control under nonstationary uncertainties, using an augmented state space and approximation techniques.

Contribution

It introduces the Markov embeddability assumption and formulates a finite-dimensional nonlinear program for control under uncertainty.

Findings

Numerical example demonstrates effective control policy performance.

Method handles nonstationary uncertainties via augmented state space.

Optimization approach uses automatic differentiation and interior-point methods.

Abstract

This article presents a constrained policy optimization approach for the optimal control of systems under nonstationary uncertainties. We introduce an assumption that we call Markov embeddability that allows us to cast the stochastic optimal control problem as a policy optimization problem over the augmented state space. Then, the infinite-dimensional policy optimization problem is approximated as a finite-dimensional nonlinear program by applying function approximation, deterministic sampling, and temporal truncation. The approximated problem is solved by using automatic differentiation and condensed-space interior-point methods. We formulate several conceptual and practical open questions regarding the asymptotic exactness of the approximation and the solution strategies for the approximated problem. As a proof of concept, we provide a numerical example demonstrating the performance of the control policy obtained by the proposed method.