Online Planning in POMDPs with Self-Improving Simulators

TL;DR

This paper introduces an online learning approach for self-improving simulators in POMDP planning, enabling faster and more efficient decision-making by adaptively balancing simulation accuracy and speed.

Contribution

It presents a novel method to learn and utilize an approximate simulator during online planning, improving efficiency over time in complex environments.

Findings

Enhanced planning efficiency with self-improving simulators

Adaptive simulation selection improves speed-accuracy trade-off

Experimental validation in large domains shows significant gains

Abstract

How can we plan efficiently in a large and complex environment when the time budget is limited? Given the original simulator of the environment, which may be computationally very demanding, we propose to learn online an approximate but much faster simulator that improves over time. To plan reliably and efficiently while the approximate simulator is learning, we develop a method that adaptively decides which simulator to use for every simulation, based on a statistic that measures the accuracy of the approximate simulator. This allows us to use the approximate simulator to replace the original simulator for faster simulations when it is accurate enough under the current context, thus trading off simulation speed and accuracy. Experimental results in two large domains show that when integrated with POMCP, our approach allows to plan with improving efficiency over time.