Learning to Explore in Diverse Reward Settings via Temporal-Difference-Error Maximization

TL;DR

This paper introduces SEE, a robust exploration method for deep reinforcement learning that maximizes TD-error to effectively handle diverse reward settings, including those that discourage exploration, without requiring hyperparameter tuning.

Contribution

The paper proposes a novel exploration technique, SEE, which maximizes TD-error with specific design choices to ensure stability and robustness across various reward environments.

Findings

SEE performs well across dense, sparse, and adverse reward settings.

The method can be integrated with off-policy algorithms without altering their optimization pipeline.

Experimental results show improved robustness and performance of Soft-Actor Critic with SEE.

Abstract

Numerous heuristics and advanced approaches have been proposed for exploration in different settings for deep reinforcement learning. Noise-based exploration generally fares well with dense-shaped rewards and bonus-based exploration with sparse rewards. However, these methods usually require additional tuning to deal with undesirable reward settings by adjusting hyperparameters and noise distributions. Rewards that actively discourage exploration, i.e., with an action cost and no other dense signal to follow, can pose a major challenge. We propose a novel exploration method, Stable Error-seeking Exploration (SEE), that is robust across dense, sparse, and exploration-adverse reward settings. To this endeavor, we revisit the idea of maximizing the TD-error as a separate objective. Our method introduces three design choices to mitigate instability caused by far-off-policy learning, the conflict of interest of maximizing the cumulative TD-error in an episodic setting, and the non-stationary nature of TD-errors. SEE can be combined with off-policy algorithms without modifying the optimization pipeline of the original objective. In our experimental analysis, we show that a Soft-Actor Critic agent with the addition of SEE performs robustly across three diverse reward settings in a variety of tasks without hyperparameter adjustments.