Random-Sampling Monte-Carlo Tree Search Methods for Cost Approximation in Long-Horizon Optimal Control

TL;DR

This paper introduces Monte-Carlo sampling methods for approximating costs in long-horizon optimal control, providing convergence analysis and demonstrating effectiveness through case studies.

Contribution

It develops and analyzes convergence properties of RS-MHP Monte-Carlo methods for cost approximation in long-horizon control problems.

Findings

Convergence in probability of cost approximation error as sample size increases

Effective application demonstrated in linear quadratic and UAV path optimization problems

Provides theoretical bounds on approximation accuracy

Abstract

In this paper, we develop Monte-Carlo based heuristic approaches to approximate the objective function in long horizon optimal control problems. In these approaches, to approximate the expectation operator in the objective function, we evolve the system state over multiple trajectories into the future while sampling the noise disturbances at each time-step, and find the average (or weighted average) of the costs along all the trajectories. We call these methods random sampling - multipath hypothesis propagation or RS-MHP. These methods (or variants) exist in the literature; however, the literature lacks results on how well these approximation strategies converge. This paper fills this knowledge gap to a certain extent. We derive convergence results for the cost approximation error from the RS-MHP methods and discuss their convergence (in probability) as the sample size increases. We consider two case studies to demonstrate the effectiveness of our methods - a) linear quadratic control problem; b) UAV path optimization problem.