Revisiting Data Augmentation in Deep Reinforcement Learning

TL;DR

This paper analyzes various data augmentation techniques in deep reinforcement learning, providing insights into their effects and proposing a regularization method to improve sample efficiency and generalization, validated across multiple domains.

Contribution

It offers a detailed analysis of data augmentation methods in DRL, introduces a novel regularization term called tangent prop, and provides principled recommendations for their effective use.

Findings

Achieves state-of-the-art performance in several environments.

Demonstrates higher sample efficiency in complex environments.

Shows improved generalization ability with the proposed methods.

Abstract

Various data augmentation techniques have been recently proposed in image-based deep reinforcement learning (DRL). Although they empirically demonstrate the effectiveness of data augmentation for improving sample efficiency or generalization, which technique should be preferred is not always clear. To tackle this question, we analyze existing methods to better understand them and to uncover how they are connected. Notably, by expressing the variance of the Q-targets and that of the empirical actor/critic losses of these methods, we can analyze the effects of their different components and compare them. We furthermore formulate an explanation about how these methods may be affected by choosing different data augmentation transformations in calculating the target Q-values. This analysis suggests recommendations on how to exploit data augmentation in a more principled way. In addition, we include a regularization term called tangent prop, previously proposed in computer vision, but whose adaptation to DRL is novel to the best of our knowledge. We evaluate our proposition and validate our analysis in several domains. Compared to different relevant baselines, we demonstrate that it achieves state-of-the-art performance in most environments and shows higher sample efficiency and better generalization ability in some complex environments.