Dynamic Subgoal-based Exploration via Bayesian Optimization

TL;DR

This paper introduces a cost-aware Bayesian optimization method for dynamic subgoal exploration in sparse-reward reinforcement learning, improving efficiency and performance in real-world navigation tasks.

Contribution

It proposes a novel Bayesian optimization framework that adaptively searches for effective subgoal strategies, considering exploration costs and environmental variability.

Findings

Outperforms existing baselines in multiple problem domains

Effectively balances exploration cost and reward acquisition

Theoretically guarantees near-optimal subgoal design asymptotically

Abstract

Reinforcement learning in sparse-reward navigation environments with expensive and limited interactions is challenging and poses a need for effective exploration. Motivated by complex navigation tasks that require real-world training (when cheap simulators are not available), we consider an agent that faces an unknown distribution of environments and must decide on an exploration strategy. It may leverage a series of training environments to improve its policy before it is evaluated in a test environment drawn from the same environment distribution. Most existing approaches focus on fixed exploration strategies, while the few that view exploration as a meta-optimization problem tend to ignore the need for cost-efficient exploration. We propose a cost-aware Bayesian optimization approach that efficiently searches over a class of dynamic subgoal-based exploration strategies. The algorithm adjusts a variety of levers -- the locations of the subgoals, the length of each episode, and the number of replications per trial -- in order to overcome the challenges of sparse rewards, expensive interactions, and noise. An experimental evaluation demonstrates that the new approach outperforms existing baselines across a number of problem domains. We also provide a theoretical foundation and prove that the method asymptotically identifies a near-optimal subgoal design.