Mixed Strategy for Constrained Stochastic Optimal Control

TL;DR

This paper introduces a randomized control strategy for constrained stochastic optimal control problems, demonstrating cost reduction via duality gap and providing efficient solution methods with practical applications.

Contribution

It extends the concept of K-randimization from finite MDPs to continuous spaces, showing its effectiveness and developing dual optimization techniques for practical problems.

Findings

Randomization can reduce expected cost in continuous stochastic control.

Duality gap quantifies the cost reduction achieved by randomization.

Efficient solution methods are demonstrated on path planning and Mars landing problems.

Abstract

Choosing control inputs randomly can result in a reduced expected cost in optimal control problems with stochastic constraints, such as stochastic model predictive control (SMPC). We consider a controller with initial randomization, meaning that the controller randomly chooses from K+1 control sequences at the beginning (called K-randimization).It is known that, for a finite-state, finite-action Markov Decision Process (MDP) with K constraints, K-randimization is sufficient to achieve the minimum cost. We found that the same result holds for stochastic optimal control problems with continuous state and action spaces.Furthermore, we show the randomization of control input can result in reduced cost when the optimization problem is nonconvex, and the cost reduction is equal to the duality gap. We then provide the necessary and sufficient conditions for the optimality of a randomized solution, and develop an efficient solution method based on dual optimization. Furthermore, in a special case with K=1 such as a joint chance-constrained problem, the dual optimization can be solved even more efficiently by root finding. Finally, we test the theories and demonstrate the solution method on multiple practical problems ranging from path planning to the planning of entry, descent, and landing (EDL) for future Mars missions.