When should agents explore?

TL;DR

This paper explores a novel approach to reinforcement learning exploration by introducing mode-switching strategies inspired by animal and human behaviors, aiming to improve adaptability and robustness.

Contribution

It proposes a new mode-switching exploration method for RL, including adaptive algorithms and analysis on Atari, addressing limitations of monolithic policies.

Findings

Effective mode-switching improves exploration diversity

Adaptive mechanisms reduce hyper-parameter tuning

Promising results on Atari benchmarks

Abstract

Exploration remains a central challenge for reinforcement learning (RL). Virtually all existing methods share the feature of a monolithic behaviour policy that changes only gradually (at best). In contrast, the exploratory behaviours of animals and humans exhibit a rich diversity, namely including forms of switching between modes. This paper presents an initial study of mode-switching, non-monolithic exploration for RL. We investigate different modes to switch between, at what timescales it makes sense to switch, and what signals make for good switching triggers. We also propose practical algorithmic components that make the switching mechanism adaptive and robust, which enables flexibility without an accompanying hyper-parameter-tuning burden. Finally, we report a promising and detailed analysis on Atari, using two-mode exploration and switching at sub-episodic time-scales.