Discrete-Time Approximations of Controlled Diffusions with Infinite Horizon Discounted and Average Cost

TL;DR

This paper develops a method to approximate optimal control policies for infinite horizon controlled diffusions using discrete-time models, ensuring near-optimality as the sampling interval decreases, thus offering a practical solution for complex control problems.

Contribution

It introduces a novel approach using discrete-time controlled Markov chains and weak convergence techniques to derive near-optimal policies for controlled diffusions, extending existing methods.

Findings

Discrete-time models can approximate controlled diffusions effectively.

Optimal policies from discrete models are near-optimal for continuous models as sampling interval shrinks.

The approach complements existing probabilistic and PDE-based methods.

Abstract

We present discrete-time approximation of optimal control policies for infinite horizon discounted/ergodic control problems for controlled diffusions in $\Rd$\,. In particular, our objective is to show near optimality of optimal policies designed from the approximating discrete-time controlled Markov chain model, for the discounted/ergodic optimal control problems, in the true controlled diffusion model (as the sampling period approaches zero). To this end, we first construct suitable discrete-time controlled Markov chain models for which one can compute optimal policies and optimal values via several methods (such as value iteration, convex analytic method, reinforcement learning etc.). Then using a weak convergence technique, we show that the optimal policy designed for the discrete-time Markov chain model is near-optimal for the controlled diffusion model as the discrete-time model approaches the continuous-time model. This provides a practical approach for finding near-optimal control policies for controlled diffusions. Our conditions complement existing results in the literature, which have been arrived at via either probabilistic or PDE based methods.