Improve Value Estimation of Q Function and Reshape Reward with Monte Carlo Tree Search

TL;DR

This paper introduces a novel Monte Carlo Tree Search-based method to improve Q value estimation and reward shaping in imperfect information games, demonstrated on Uno, leading to better performance than traditional approaches.

Contribution

The paper proposes a new algorithm combining Monte Carlo Tree Search with Q learning to reduce overestimation and reshape rewards in imperfect information games.

Findings

Our method outperforms traditional algorithms in Uno.

Performance gains increase with more players, indicating higher difficulty.

The approach is generalizable to other Q value estimation algorithms.

Abstract

Reinforcement learning has achieved remarkable success in perfect information games such as Go and Atari, enabling agents to compete at the highest levels against human players. However, research in reinforcement learning for imperfect information games has been relatively limited due to the more complex game structures and randomness. Traditional methods face challenges in training and improving performance in imperfect information games due to issues like inaccurate Q value estimation and reward sparsity. In this paper, we focus on Uno, an imperfect information game, and aim to address these problems by reducing Q value overestimation and reshaping reward function. We propose a novel algorithm that utilizes Monte Carlo Tree Search to average the value estimations in Q function. Even though we choose Double Deep Q Learning as the foundational framework in this paper, our method can be generalized and used in any algorithm which needs Q value estimation, such as the Actor-Critic. Additionally, we employ Monte Carlo Tree Search to reshape the reward structure in the game environment. We compare our algorithm with several traditional methods applied to games such as Double Deep Q Learning, Deep Monte Carlo and Neural Fictitious Self Play, and the experiments demonstrate that our algorithm consistently outperforms these approaches, especially as the number of players in Uno increases, indicating a higher level of difficulty.