Simple Noisy Environment Augmentation for Reinforcement Learning

TL;DR

This paper introduces generic noise-based environment augmentation wrappers for reinforcement learning, enhancing exploration and data diversity, with experimental validation across multiple algorithms and environments, and provides open-source tools for practical use.

Contribution

The paper proposes a set of broad, noise-based augmentation wrappers for RL environments, including novel techniques and a noise rate hyperparameter, applicable to various algorithms and environments.

Findings

Augmentation wrappers improve RL performance across algorithms.

Noise rate hyperparameter effectively controls noise injection frequency.

Experimental results demonstrate enhanced exploration and training stability.

Abstract

Data augmentation is a widely used technique for improving model performance in machine learning, particularly in computer vision and natural language processing. Recently, there has been increasing interest in applying augmentation techniques to reinforcement learning (RL) problems, with a focus on image-based augmentation. In this paper, we explore a set of generic wrappers designed to augment RL environments with noise and encourage agent exploration and improve training data diversity which are applicable to a broad spectrum of RL algorithms and environments. Specifically, we concentrate on augmentations concerning states, rewards, and transition dynamics and introduce two novel augmentation techniques. In addition, we introduce a noise rate hyperparameter for control over the frequency of noise injection. We present experimental results on the impact of these wrappers on return using three popular RL algorithms, Soft Actor-Critic (SAC), Twin Delayed DDPG (TD3), and Proximal Policy Optimization (PPO), across five MuJoCo environments. To support the choice of augmentation technique in practice, we also present analysis that explores the performance these techniques across environments. Lastly, we publish the wrappers in our noisyenv repository for use with gym environments.