Mirror descent for constrained stochastic control problems

TL;DR

This paper develops continuous-time mirror descent methods for constrained stochastic control problems, demonstrating linear and exponential convergence under convexity conditions, and addresses key analytical challenges with PDE and Sobolev space techniques.

Contribution

It introduces a novel mirror descent framework for stochastic control with convex action spaces and provides convergence analysis under convexity assumptions.

Findings

Mirror descent converges linearly when the Hamiltonian is uniformly convex.

Exponential convergence occurs if the Hamiltonian is strongly convex relative to a Bregman divergence.

The paper overcomes analytical challenges using PDE estimates and the performance difference lemma.

Abstract

Mirror descent is a well established tool for solving convex optimization problems with convex constraints. This article introduces continuous-time mirror descent dynamics for approximating optimal Markov controls for stochastic control problems with the action space being bounded and convex. We show that if the Hamiltonian is uniformly convex in its action variable then mirror descent converges linearly while if it is uniformly strongly convex relative to an appropriate Bregman divergence, then the mirror flow converges exponentially. The two fundamental difficulties that must be overcome to prove such results are: first, the inherent lack of convexity of the map from Markov controls to the corresponding value function. Second, maintaining sufficient regularity of the value function and the Markov controls along the mirror descent updates. The first issue is handled using the performance difference lemma, while the second using careful Sobolev space estimates for the solutions of the associated linear PDEs.