Linear programming-based solution methods for constrained partially observable Markov decision processes

TL;DR

This paper introduces linear programming-based approximation methods for solving constrained partially observable Markov decision processes (CPOMDPs), demonstrating their effectiveness and flexibility through extensive numerical experiments.

Contribution

It presents a novel LP-based approach for approximating solutions to CPOMDPs, including the integration of deterministic policy constraints and analysis of their effects.

Findings

LP methods effectively approximate CPOMDP solutions.

Deterministic constraints have minimal impact on finite horizon rewards.

Deterministic policies yield lower rewards in infinite horizon problems.

Abstract

Constrained partially observable Markov decision processes (CPOMDPs) have been used to model various real-world phenomena. However, they are notoriously difficult to solve to optimality, and there exist only a few approximation methods for obtaining high-quality solutions. In this study, grid-based approximations are used in combination with linear programming (LP) models to generate approximate policies for CPOMDPs. A detailed numerical study is conducted with six CPOMDP problem instances considering both their finite and infinite horizon formulations. The quality of approximation algorithms for solving unconstrained POMDP problems is established through a comparative analysis with exact solution methods. Then, the performance of the LP-based CPOMDP solution approaches for varying budget levels is evaluated. Finally, the flexibility of LP-based approaches is demonstrated by applying deterministic policy constraints, and a detailed investigation into their impact on rewards and CPU run time is provided. For most of the finite horizon problems, deterministic policy constraints are found to have little impact on expected reward, but they introduce a significant increase to CPU run time. For infinite horizon problems, the reverse is observed: deterministic policies tend to yield lower expected total rewards than their stochastic counterparts, but the impact of deterministic constraints on CPU run time is negligible in this case. Overall, these results demonstrate that LP models can effectively generate approximate policies for both finite and infinite horizon problems while providing the flexibility to incorporate various additional constraints into the underlying model.