A Bayesian Approach to Online Planning

TL;DR

This paper introduces a Bayesian online planning method that uses uncertainty estimates from neural networks to improve decision-making in complex environments, providing theoretical guarantees and empirical validation.

Contribution

It develops a Bayesian planning algorithm with finite-time regret bounds and demonstrates its effectiveness over existing methods in challenging environments.

Findings

Bayesian approach improves search efficiency when uncertainty estimates are accurate.

Proposed algorithms outperform traditional methods in ProcGen Maze and Leaper environments.

Uncertainty estimation quality significantly impacts planning performance.

Abstract

The combination of Monte Carlo tree search and neural networks has revolutionized online planning. As neural network approximations are often imperfect, we ask whether uncertainty estimates about the network outputs could be used to improve planning. We develop a Bayesian planning approach that facilitates such uncertainty quantification, inspired by classical ideas from the meta-reasoning literature. We propose a Thompson sampling based algorithm for searching the tree of possible actions, for which we prove the first (to our knowledge) finite time Bayesian regret bound, and propose an efficient implementation for a restricted family of posterior distributions. In addition we propose a variant of the Bayes-UCB method applied to trees. Empirically, we demonstrate that on the ProcGen Maze and Leaper environments, when the uncertainty estimates are accurate but the neural network output is inaccurate, our Bayesian approach searches the tree much more effectively. In addition, we investigate whether popular uncertainty estimation methods are accurate enough to yield significant gains in planning. Our code is available at: https://github.com/nirgreshler/bayesian-online-planning.