Combining Offline Models and Online Monte-Carlo Tree Search for Planning from Scratch

TL;DR

This paper presents a novel planning approach that combines offline learned Predictive State Representations with online Monte-Carlo tree search to effectively handle model uncertainty in stochastic, partially observable environments, demonstrating improved scalability and convergence.

Contribution

It introduces a method that learns an offline PSR model from scratch and integrates it with online MCTS, advancing planning under model uncertainty without prior environment knowledge.

Findings

Proven convergence of the proposed approach.

Outperforms state-of-the-art methods in scalability.

Successfully applied to the RockSample problem.

Abstract

Planning in stochastic and partially observable environments is a central issue in artificial intelligence. One commonly used technique for solving such a problem is by constructing an accurate model firstly. Although some recent approaches have been proposed for learning optimal behaviour under model uncertainty, prior knowledge about the environment is still needed to guarantee the performance of the proposed algorithms. With the benefits of the Predictive State Representations~(PSRs) approach for state representation and model prediction, in this paper, we introduce an approach for planning from scratch, where an offline PSR model is firstly learned and then combined with online Monte-Carlo tree search for planning with model uncertainty. By comparing with the state-of-the-art approach of planning with model uncertainty, we demonstrated the effectiveness of the proposed approaches along with the proof of their convergence. The effectiveness and scalability of our proposed approach are also tested on the RockSample problem, which are infeasible for the state-of-the-art BA-POMDP based approaches.