Search Inspired Exploration in Reinforcement Learning

TL;DR

This paper introduces SIERL, a novel exploration method for reinforcement learning that actively guides agents using search-inspired sub-goal selection, improving performance in sparse-reward environments.

Contribution

SIERL's key innovation is a systematic sub-goal selection mechanism based on search principles, enhancing exploration efficiency and effectiveness in sparse-reward RL tasks.

Findings

SIERL outperforms baseline methods in challenging environments.

SIERL achieves better task success rates and state reachability.

The method generalizes well to arbitrary states.

Abstract

Exploration in environments with sparse rewards remains a fundamental challenge in reinforcement learning (RL). Existing approaches such as curriculum learning and Go-Explore often rely on hand-crafted heuristics, while curiosity-driven methods risk converging to suboptimal policies. We propose Search-Inspired Exploration in Reinforcement Learning (SIERL), a novel method that actively guides exploration by setting sub-goals based on the agent's learning progress. At the beginning of each episode, SIERL chooses a sub-goal from the \textit{frontier} (the boundary of the agent's known state space), before the agent continues exploring toward the main task objective. The key contribution of our method is the sub-goal selection mechanism, which provides state-action pairs that are neither overly familiar nor completely novel. Thus, it assures that the frontier is expanded systematically and that the agent is capable of reaching any state within it. Inspired by search, sub-goals are prioritized from the frontier based on estimates of cost-to-come and cost-to-go, effectively steering exploration towards the most informative regions. In experiments on challenging sparse-reward environments, SIERL outperforms dominant baselines in both achieving the main task goal and generalizing to reach arbitrary states in the environment.