Experimental design for Partially Observed Markov Decision Processes

TL;DR

This paper develops an experimental design framework for Partially Observed Markov Decision Processes, optimizing data collection to improve parameter estimation using dynamic programming and Fisher Information maximization.

Contribution

It introduces a novel algorithm combining dynamic programming and Fisher Information to design optimal experiments for POMDPs, with applications to neural models and PCR growth dynamics.

Findings

Effective control policies for POMDPs are derived.

Application to neural and PCR models demonstrates practical utility.

Online updating of control policies enhances adaptability.

Abstract

This paper deals with the question of how to most effectively conduct experiments in Partially Observed Markov Decision Processes so as to provide data that is most informative about a parameter of interest. Methods from Markov decision processes, especially dynamic programming, are introduced and then used in an algorithm to maximize a relevant Fisher Information. The algorithm is then applied to two POMDP examples. The methods developed can also be applied to stochastic dynamical systems, by suitable discretization, and we consequently show what control policies look like in the Morris-Lecar Neuron model, and simulation results are presented. We discuss how parameter dependence within these methods can be dealt with by the use of priors, and develop tools to update control policies online. This is demonstrated in another stochastic dynamical system describing growth dynamics of DNA template in a PCR model.