# Action Guidance with MCTS for Deep Reinforcement Learning

## TL;DR

This paper introduces a framework combining Monte Carlo tree search with deep reinforcement learning to enhance sample efficiency, demonstrating faster learning and better policies in multi-agent Pommerman with sparse rewards.

## Contribution

It presents a novel integration of non-expert MCTS demonstrators into deep RL, improving learning speed and policy quality in complex multi-agent environments.

## Key findings

- Faster learning compared to vanilla deep RL.
- Converges to better policies in Pommerman.
- Effective use of non-expert demonstrators.

## Abstract

Deep reinforcement learning has achieved great successes in recent years, however, one main challenge is the sample inefficiency. In this paper, we focus on how to use action guidance by means of a non-expert demonstrator to improve sample efficiency in a domain with sparse, delayed, and possibly deceptive rewards: the recently-proposed multi-agent benchmark of Pommerman. We propose a new framework where even a non-expert simulated demonstrator, e.g., planning algorithms such as Monte Carlo tree search with a small number rollouts, can be integrated within asynchronous distributed deep reinforcement learning methods. Compared to a vanilla deep RL algorithm, our proposed methods both learn faster and converge to better policies on a two-player mini version of the Pommerman game.