Continuous Policy and Value Iteration for Stochastic Control Problems and Its Convergence

TL;DR

This paper presents a continuous policy-value iteration method for stochastic control problems, leveraging Langevin dynamics to enable simultaneous updates of value functions and controls, applicable to both regularized and classical problems with proven convergence.

Contribution

It introduces a novel continuous policy-value iteration algorithm using Langevin dynamics, applicable to a broad class of stochastic control problems, with convergence guarantees.

Findings

Convergence to optimal control under monotonicity of the Hamiltonian.

Framework accommodates entropy-regularized and classical control problems.

Enables distribution sampling and non-convex learning techniques.

Abstract

We introduce a continuous policy-value iteration algorithm where the approximations of the value function of a stochastic control problem and the optimal control are simultaneously updated through Langevin-type dynamics. This framework applies to both the entropy-regularized relaxed control problems and the classical control problems, with infinite horizon. We establish policy improvement and demonstrate convergence to the optimal control under the monotonicity condition of the Hamiltonian. By utilizing Langevin-type stochastic differential equations for continuous updates along the policy iteration direction, our approach enables the use of distribution sampling and non-convex learning techniques in machine learning to optimize the value function and identify the optimal control simultaneously.