Exploration by Random Reward Perturbation

TL;DR

This paper presents Random Reward Perturbation (RRP), a simple yet effective exploration method for reinforcement learning that improves policy diversity and performance by adding zero-mean reward noise, compatible with existing strategies.

Contribution

The paper introduces RRP, a novel reward perturbation technique that enhances exploration in RL, providing theoretical insights and demonstrating empirical improvements in sample efficiency and task performance.

Findings

RRP improves exploration and policy diversity.

RRP enhances performance of PPO and SAC algorithms.

RRP achieves higher sample efficiency across tasks.

Abstract

We introduce Random Reward Perturbation (RRP), a novel exploration strategy for reinforcement learning (RL). Our theoretical analyses demonstrate that adding zero-mean noise to environmental rewards effectively enhances policy diversity during training, thereby expanding the range of exploration. RRP is fully compatible with the action-perturbation-based exploration strategies, such as $\epsilon$-greedy, stochastic policies, and entropy regularization, providing additive improvements to exploration effects. It is general, lightweight, and can be integrated into existing RL algorithms with minimal implementation effort and negligible computational overhead. RRP establishes a theoretical connection between reward shaping and noise-driven exploration, highlighting their complementary potential. Experiments show that RRP significantly boosts the performance of Proximal Policy Optimization and Soft Actor-Critic, achieving higher sample efficiency and escaping local optima across various tasks, under both sparse and dense reward scenarios.