Limited depth bandit-based strategy for Monte Carlo planning in continuous action spaces

TL;DR

This paper introduces LD-HOO, a limited depth variant of HOO, improving efficiency in continuous action space search trees for optimal control, with comparable regret and better speed and memory use.

Contribution

The paper proposes LD-HOO, a novel limited depth algorithm for continuous action bandits, with theoretical regret guarantees and practical application in Monte Carlo tree search for control.

Findings

LD-HOO matches HOO's asymptotic regret

LD-HOO is faster and more memory-efficient

Effective in several optimal control problems

Abstract

This paper addresses the problem of optimal control using search trees. We start by considering multi-armed bandit problems with continuous action spaces and propose LD-HOO, a limited depth variant of the hierarchical optimistic optimization (HOO) algorithm. We provide a regret analysis for LD-HOO and show that, asymptotically, our algorithm exhibits the same cumulative regret as the original HOO while being faster and more memory efficient. We then propose a Monte Carlo tree search algorithm based on LD-HOO for optimal control problems and illustrate the resulting approach's application in several optimal control problems.