Approximation Methods for Partially Observed Markov Decision Processes (POMDPs)

TL;DR

This survey reviews the origins, theory, and approximation methods for finite-state POMDPs, emphasizing their computational challenges and recent research directions in control under uncertainty.

Contribution

It provides a comprehensive overview of POMDPs, focusing on approximation techniques and recent research directions, with essential background on MDPs and HMMs.

Findings

Approximation methods are crucial for handling large state spaces in POMDPs.

Exact solutions are computationally intensive, necessitating efficient approximations.

Emerging research directions aim to improve scalability and solution quality.

Abstract

POMDPs are useful models for systems where the true underlying state is not known completely to an outside observer; the outside observer incompletely knows the true state of the system, and observes a noisy version of the true system state. When the number of system states is large in a POMDP that often necessitates the use of approximation methods to obtain near optimal solutions for control. This survey is centered around the origins, theory, and approximations of finite-state POMDPs. In order to understand POMDPs, it is required to have an understanding of finite-state Markov Decision Processes (MDPs) in \autoref{mdp} and Hidden Markov Models (HMMs) in \autoref{hmm}. For this background theory, I provide only essential details on MDPs and HMMs and leave longer expositions to textbook treatments before diving into the main topics of POMDPs. Once the required background is covered, the POMDP is introduced in \autoref{pomdp}. The origins of the POMDP are explained in the classical papers section \autoref{classical}. Once the high computational requirements are understood from the exact methodological point of view, the main approximation methods are surveyed in \autoref{approximations}. Then, I end the survey with some new research directions in \autoref{conclusion}.