Anytime Incremental $\rho$POMDP Planning in Continuous Spaces

TL;DR

This paper introduces $ ho$POMCPOW, an anytime incremental solver for continuous-space $ ho$POMDPs that dynamically refines beliefs and reduces computational costs, improving efficiency and solution quality in decision-making tasks.

Contribution

The paper presents a novel incremental belief update method for $ ho$POMDPs, enabling dynamic refinement and formal guarantees of improvement over time.

Findings

Reduces belief update computational costs by orders of magnitude.

Outperforms existing solvers in efficiency and solution quality.

Provides formal guarantees of improvement over time.

Abstract

Partially Observable Markov Decision Processes (POMDPs) provide a robust framework for decision-making under uncertainty in applications such as autonomous driving and robotic exploration. Their extension, $\rho$POMDPs, introduces belief-dependent rewards, enabling explicit reasoning about uncertainty. Existing online $\rho$POMDP solvers for continuous spaces rely on fixed belief representations, limiting adaptability and refinement - critical for tasks such as information-gathering. We present $\rho$POMCPOW, an anytime solver that dynamically refines belief representations, with formal guarantees of improvement over time. To mitigate the high computational cost of updating belief-dependent rewards, we propose a novel incremental computation approach. We demonstrate its effectiveness for common entropy estimators, reducing computational cost by orders of magnitude. Experimental results show that $\rho$POMCPOW outperforms state-of-the-art solvers in both efficiency and solution quality.