Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control

TL;DR

This paper introduces Monte Carlo Beam Search (MCBS), a hybrid exploration method combining beam search and Monte Carlo rollouts with actor-critic algorithms to improve policy convergence and sample efficiency in continuous control tasks.

Contribution

The paper presents MCBS, a novel hybrid exploration technique that enhances actor-critic reinforcement learning by integrating structured look-ahead search with existing algorithms like TD3.

Findings

MCBS outperforms TD3, SAC, PPO, and A2C in sample efficiency and convergence speed.

MCBS achieves 90% of maximum reward in fewer timesteps compared to baseline methods.

Hyperparameter analysis and adaptive strategies improve MCBS performance in complex environments.

Abstract

Actor-critic methods, like Twin Delayed Deep Deterministic Policy Gradient (TD3), depend on basic noise-based exploration, which can result in less than optimal policy convergence. In this study, we introduce Monte Carlo Beam Search (MCBS), a new hybrid method that combines beam search and Monte Carlo rollouts with TD3 to improve exploration and action selection. MCBS produces several candidate actions around the policy's output and assesses them through short-horizon rollouts, enabling the agent to make better-informed choices. We test MCBS across various continuous-control benchmarks, including HalfCheetah-v4, Walker2d-v5, and Swimmer-v5, showing enhanced sample efficiency and performance compared to standard TD3 and other baseline methods like SAC, PPO, and A2C. Our findings emphasize MCBS's capability to enhance policy learning through structured look-ahead search while ensuring computational efficiency. Additionally, we offer a detailed analysis of crucial hyperparameters, such as beam width and rollout depth, and explore adaptive strategies to optimize MCBS for complex control tasks. Our method shows a higher convergence rate across different environments compared to TD3, SAC, PPO, and A2C. For instance, we achieved 90% of the maximum achievable reward within around 200 thousand timesteps compared to 400 thousand timesteps for the second-best method.