Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration

TL;DR

This paper introduces a novel exploration method that decouples exploration from policy optimization, using a tree search guided by uncertainty, leading to more efficient exploration and state coverage in complex environments.

Contribution

It proposes a new approach that bypasses reinforcement learning during exploration, enabling more efficient discovery and policy distillation for hard exploration tasks.

Findings

Explores an order of magnitude more efficiently than intrinsic motivation baselines.

Achieves state-of-the-art performance on Montezuma's Revenge, Pitfall!, and Venture.

Successfully solves high-dimensional continuous tasks from images without demonstrations.

Abstract

The process of discovery requires active exploration -- the act of collecting new and informative data. However, efficient autonomous exploration remains a major unsolved problem. The dominant paradigm addresses this challenge by using Reinforcement Learning (RL) to train agents with intrinsic motivation, maximizing a composite objective of extrinsic and intrinsic rewards. We suggest that this approach incurs unnecessary overhead: while policy optimization is necessary for precise task execution, employing such machinery solely to expand state coverage may be inefficient. In this paper, we propose a new approach that explicitly decouples exploration from policy optimization and bypasses RL entirely during the exploration phase. Our method uses a tree-search strategy inspired by the Go-With-The-Winner algorithm, paired with a measure of uncertainty to systematically drive exploration. By removing the overhead of policy optimization, our approach explores an order of magnitude more efficiently than standard intrinsic motivation baselines on hard exploration benchmarks. Further, we demonstrate that the trajectories discovered during exploration can be distilled into deployable policies using existing supervised backward learning algorithms, achieving state-of-the-art performance by a wide margin on Montezuma's Revenge, Pitfall!, and Venture without relying on domain-specific knowledge. Finally, we demonstrate the generality of our framework in high-dimensional continuous action spaces by solving the MuJoCo Adroit dexterous manipulation and AntMaze tasks in a sparse-reward setting, directly from image observations and without expert demonstrations or offline datasets. To the best of our knowledge, this has not been achieved before for the Adroit tasks.