Online algorithms for POMDPs with continuous state, action, and observation spaces

TL;DR

This paper addresses the challenge of applying online POMDP algorithms to continuous spaces by proposing two new algorithms, POMCPOW and PFT-DPW, which use weighted particle filtering to improve policy quality.

Contribution

The paper introduces POMCPOW and PFT-DPW algorithms that overcome belief collapse issues in continuous POMDPs by employing weighted particle filtering techniques.

Findings

POMCPOW and PFT-DPW outperform previous methods in continuous POMDPs.

Weighted particle filtering prevents belief collapse in search trees.

Simulation results demonstrate improved success in complex continuous problems.

Abstract

Online solvers for partially observable Markov decision processes have been applied to problems with large discrete state spaces, but continuous state, action, and observation spaces remain a challenge. This paper begins by investigating double progressive widening (DPW) as a solution to this challenge. However, we prove that this modification alone is not sufficient because the belief representations in the search tree collapse to a single particle causing the algorithm to converge to a policy that is suboptimal regardless of the computation time. This paper proposes and evaluates two new algorithms, POMCPOW and PFT-DPW, that overcome this deficiency by using weighted particle filtering. Simulation results show that these modifications allow the algorithms to be successful where previous approaches fail.