Value-Directed Sampling Methods for POMDPs

TL;DR

This paper introduces value-directed sampling methods for POMDPs, providing error bounds and an adaptive sampling procedure to improve decision-making accuracy with particle filtering.

Contribution

It develops a novel approach that dynamically allocates sampling effort in particle filtering for POMDPs based on value considerations, with theoretical error bounds and empirical validation.

Findings

Error bounds on decision quality with importance sampling

An adaptive sampling procedure reduces unnecessary computation

Empirical results show improved policy differentiation

Abstract

We consider the problem of approximate belief-state monitoring using particle filtering for the purposes of implementing a policy for a partially-observable Markov decision process (POMDP). While particle filtering has become a widely-used tool in AI for monitoring dynamical systems, rather scant attention has been paid to their use in the context of decision making. Assuming the existence of a value function, we derive error bounds on decision quality associated with filtering using importance sampling. We also describe an adaptive procedure that can be used to dynamically determine the number of samples required to meet specific error bounds. Empirical evidence is offered supporting this technique as a profitable means of directing sampling effort where it is needed to distinguish policies.