Lipschitz Lifelong Monte Carlo Tree Search for Mastering Non-Stationary Tasks

TL;DR

LiZero introduces a Lipschitz-based adaptive UCT for MCTS, enabling efficient lifelong planning in non-stationary environments, significantly improving convergence speed over existing methods.

Contribution

The paper proposes LiZero with a novel adaptive UCT that transfers knowledge across tasks considering Lipschitz continuity, advancing lifelong MCTS in non-stationary settings.

Findings

LiZero achieves 3-4x faster convergence to optimal rewards.

The adaptive UCT effectively transfers knowledge between tasks.

The algorithms are computationally efficient with characterized error bounds.

Abstract

Monte Carlo Tree Search (MCTS) has proven highly effective in solving complex planning tasks by balancing exploration and exploitation using Upper Confidence Bound for Trees (UCT). However, existing work have not considered MCTS-based lifelong planning, where an agent faces a non-stationary series of tasks -- e.g., with varying transition probabilities and rewards -- that are drawn sequentially throughout the operational lifetime. This paper presents LiZero for Lipschitz lifelong planning using MCTS. We propose a novel concept of adaptive UCT (aUCT) to transfer knowledge from a source task to the exploration/exploitation of a new task, depending on both the Lipschitz continuity between tasks and the confidence of knowledge in in Monte Carlo action sampling. We analyze LiZero's acceleration factor in terms of improved sampling efficiency and also develop efficient algorithms to compute aUCT in an online fashion by both data-driven and model-based approaches, whose sampling complexity and error bounds are also characterized. Experiment results show that LiZero significantly outperforms existing MCTS and lifelong learning baselines in terms of much faster convergence (3$\sim$4x) to optimal rewards. Our results highlight the potential of LiZero to advance decision-making and planning in dynamic real-world environments.