Enhancing Reinforcement Learning Through Guided Search

TL;DR

This paper explores using Monte Carlo Tree Search as a guiding policy to enhance reinforcement learning performance, demonstrating significant improvements on the Atari 100k benchmark.

Contribution

It introduces a novel approach of integrating MCTS as a guide in RL agents, inspired by Offline RL strategies, to improve performance in off-policy settings.

Findings

MCTS-guided RL outperforms standalone methods.

Significant performance gains on Atari 100k benchmark.

Guided search effectively reduces policy errors.

Abstract

With the aim of improving performance in Markov Decision Problem in an Off-Policy setting, we suggest taking inspiration from what is done in Offline Reinforcement Learning (RL). In Offline RL, it is a common practice during policy learning to maintain proximity to a reference policy to mitigate uncertainty, reduce potential policy errors, and help improve performance. We find ourselves in a different setting, yet it raises questions about whether a similar concept can be applied to enhance performance ie, whether it is possible to find a guiding policy capable of contributing to performance improvement, and how to incorporate it into our RL agent. Our attention is particularly focused on algorithms based on Monte Carlo Tree Search (MCTS) as a guide.MCTS renowned for its state-of-the-art capabilities across various domains, catches our interest due to its ability to converge to equilibrium in single-player and two-player contexts. By harnessing the power of MCTS as a guide for our RL agent, we observed a significant performance improvement, surpassing the outcomes achieved by utilizing each method in isolation. Our experiments were carried out on the Atari 100k benchmark.