Convex Analytic Method Revisited: Further Optimality Results and Performance of Deterministic Policies in Average Cost Stochastic Control

TL;DR

This paper revisits the convex analytic method in stochastic control, providing new existence results for certain Markov models, conditions for deterministic policy optimality, and demonstrating the density of deterministic policy performance.

Contribution

It offers new existence results for Markov models without weak continuity, conditions for deterministic policy optimality, and shows performance density of deterministic policies in average cost control.

Findings

Existence of solutions for strongly continuous transition kernels.

Conditions ensuring deterministic stationary policy optimality.

Density of deterministic policy performance in randomized policies.

Abstract

The convex analytic method has proved to be a very versatile method for the study of infinite horizon average cost optimal stochastic control problems. In this paper, we revisit the convex analytic method and make three primary contributions: (i) We present an existence result for controlled Markov models that lack weak continuity of the transition kernel but are strongly continuous in the action variable for every fixed state variable. (ii) For average cost stochastic control problems in standard Borel spaces, while existing results establish the optimality of stationary (possibly randomized) policies, few results are available on the optimality of deterministic policies. We review existing results and present further conditions under which an average cost optimal stochastic control problem admits optimal solutions that are deterministic stationary. (iii) We establish conditions under which the performance under stationary deterministic (and also quantized) policies is dense in the set of performance values under randomized stationary policies.