Reannealing of Decaying Exploration Based On Heuristic Measure in Deep Q-Network

TL;DR

This paper introduces a simple reannealing-based exploration strategy for deep Q-networks that activates exploration only when the agent appears stuck, potentially speeding up training and improving policies.

Contribution

It proposes a novel, easy-to-implement reannealing method that dynamically adjusts exploration based on heuristic feedback in deep reinforcement learning.

Findings

Potential to accelerate training

Leads to better policies

Effective in illustrative case study

Abstract

Existing exploration strategies in reinforcement learning (RL) often either ignore the history or feedback of search, or are complicated to implement. There is also a very limited literature showing their effectiveness over diverse domains. We propose an algorithm based on the idea of reannealing, that aims at encouraging exploration only when it is needed, for example, when the algorithm detects that the agent is stuck in a local optimum. The approach is simple to implement. We perform an illustrative case study showing that it has potential to both accelerate training and obtain a better policy.