Neighboring State-based Exploration for Reinforcement Learning

TL;DR

This paper introduces neighboring state-based exploration algorithms for reinforcement learning, demonstrating significant performance improvements over baseline methods by focusing on nearby states during exploration.

Contribution

The paper proposes two novel algorithms for neighboring state-based exploration, with one method, ${\rho}$-explore, outperforming standard Double DQN in discrete environments.

Findings

${\rho}$-explore outperforms Double DQN by 49% in Eval Reward Return

Neighboring state-based exploration improves early-stage decision-making

Proposed algorithms effectively leverage local state information

Abstract

Reinforcement Learning is a powerful tool to model decision-making processes. However, it relies on an exploration-exploitation trade-off that remains an open challenge for many tasks. In this work, we study neighboring state-based, model-free exploration led by the intuition that, for an early-stage agent, considering actions derived from a bounded region of nearby states may lead to better actions when exploring. We propose two algorithms that choose exploratory actions based on a survey of nearby states, and find that one of our methods, ${\rho}$-explore, consistently outperforms the Double DQN baseline in an discrete environment by 49% in terms of Eval Reward Return.