Numerical method to solve impulse control problems for partially observed piecewise deterministic Markov processes

TL;DR

This paper develops a numerical method for solving impulse control problems in partially observed PDMPs by discretizing the state space and approximating the POMDP, addressing challenges from random jumps and hidden information.

Contribution

It introduces a novel discretization approach for POMDPs derived from PDMPs with hidden jumps, enabling practical numerical solutions.

Findings

Effective discretization grids are constructed for complex state spaces.

The method accurately approximates the value function in simulations.

Discretization errors are carefully analyzed and controlled.

Abstract

Designing efficient and rigorous numerical methods for sequential decision-making under uncertainty is a difficult problem that arises in many applications frameworks. In this paper we focus on the numerical solution of a subclass of impulse control problem for piecewise deterministic Markov process (PDMP) when the jump times are hidden. We first state the problem as a partially observed Markov decision process (POMDP) on a continuous state space and with controlled transition kernels corresponding to some specific skeleton chains of the PDMP. Then we proceed to build a numerically tractable approximation of the POMDP by tailor-made discretizations of the state spaces. The main difficulty in evaluating the discretization error comes from the possible random jumps of the PDMP between consecutive epochs of the POMDP and requires special care. Finally we discuss the practical construction of discretization grids and illustrate our method on simulations.