Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

TL;DR

This paper offers a novel approach to partially observed stochastic control, establishing new existence, ergodicity, and approximation results without relying on belief-state reduction, applicable to both discounted and average cost criteria.

Contribution

It introduces an alternative framework for POMDPs using infinite-dimensional product spaces, providing new conditions for optimal policy existence and convergence without belief reduction.

Findings

Established conditions for optimal policy existence under discounted and average costs.

Proved near optimality of finite window policies via quantization techniques.

Derived new conditions for invariant measure existence in nonlinear filtering.

Abstract

We present an alternative view for the study of optimal control of partially observed Markov Decision Processes (POMDPs). We first revisit the traditional (and by now standard) separated-design method of reducing the problem to fully observed MDPs (belief-MDPs), and present conditions for the existence of optimal policies. Then, rather than working with this standard method, we define a Markov chain taking values in an infinite dimensional product space with the history process serving as the controlled state process and a further refinement in which the control actions and the state process are causally conditionally independent given the measurement/information process. We provide new sufficient conditions for the existence of optimal control policies under the discounted cost and average cost infinite horizon criteria. For the discounted cost setup, we establish near optimality of finite window policies via a direct argument involving near optimality of quantized approximations for MDPs under weak Feller continuity, where finite truncations of memory can be viewed as quantizations of infinite memory with a uniform diameter in each finite window restriction under the product metric. For the average cost setup, we provide new existence conditions and also a general approach on how to initialize the randomness which we show to establish convergence to optimal cost. In the control-free case, our analysis leads to new and weak conditions for the existence and uniqueness of invariant probability measures for nonlinear filter processes.