Partially Observed Optimal Stochastic Control: Regularity, Optimality, Approximations, and Learning

TL;DR

This paper reviews recent advances in the optimal control of partially observed Markov decision processes, focusing on regularity, existence, approximation methods, and reinforcement learning convergence results.

Contribution

It provides a comprehensive overview of regularity conditions, approximation techniques, and recent RL convergence results for POMDPs, integrating theory and practical algorithms.

Findings

Established regularity and continuity conditions for POMDPs.

Proved existence of optimal policies under certain conditions.

Demonstrated convergence of RL algorithms to near-optimal solutions.

Abstract

In this review/tutorial article, we present recent progress on optimal control of partially observed Markov Decision Processes (POMDPs). We first present regularity and continuity conditions for POMDPs and their belief-MDP reductions, where these constitute weak Feller and Wasserstein regularity and controlled filter stability. These are then utilized to arrive at existence results on optimal policies for both discounted and average cost problems, and regularity of value functions. Then, we study rigorous approximation results involving quantization based finite model approximations as well as finite window approximations under controlled filter stability. Finally, we present several recent reinforcement learning theoretic results which rigorously establish convergence to near optimality under both criteria.