Tail-Risk-Safe Monte Carlo Tree Search under PAC-Level Guarantees

TL;DR

This paper introduces two novel Monte Carlo Tree Search methods, CVaR-MCTS and W-MCTS, that provide rigorous tail-risk guarantees for decision-making in high-stakes scenarios, addressing limitations of existing safety-aware approaches.

Contribution

The paper develops PAC-guaranteed tail-risk-safe MCTS algorithms by embedding CVaR and Wasserstein ambiguity sets, offering explicit control over extreme adverse outcomes.

Findings

Both methods outperform baselines in simulated environments.

They achieve robust tail-risk guarantees with improved rewards.

The approaches provide theoretical PAC tail-safety guarantees.

Abstract

Making decisions with respect to just the expected returns in Monte Carlo Tree Search (MCTS) cannot account for the potential range of high-risk, adverse outcomes associated with a decision. To this end, safety-aware MCTS often consider some constrained variants -- by introducing some form of mean risk measures or hard cost thresholds. These approaches fail to provide rigorous tail-safety guarantees with respect to extreme or high-risk outcomes (denoted as tail-risk), potentially resulting in serious consequence in high-stake scenarios. This paper addresses the problem by developing two novel solutions. We first propose CVaR-MCTS, which embeds a coherent tail risk measure, Conditional Value-at-Risk (CVaR), into MCTS. Our CVaR-MCTS with parameter $\alpha$ achieves explicit tail-risk control over the expected loss in the "worst $(1-\alpha)\%$ scenarios." Second, we further address the estimation bias of tail-risk due to limited samples. We propose Wasserstein-MCTS (or W-MCTS) by introducing a first-order Wasserstein ambiguity set $\mathcal{P}_{\varepsilon_{s}}(s,a)$ with radius $\varepsilon_{s}$ to characterize the uncertainty in tail-risk estimates. We prove PAC tail-safety guarantees for both CVaR-MCTS and W-MCTS and establish their regret. Evaluations on diverse simulated environments demonstrate that our proposed methods outperform existing baselines, effectively achieving robust tail-risk guarantees with improved rewards and stability.