Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning

TL;DR

This paper introduces an adaptive scheduling framework for data augmentation in deep reinforcement learning, improving sample efficiency and generalization by optimally timing augmentation and distillation processes.

Contribution

It proposes a novel framework that automatically schedules data augmentation and distillation in RL, including a stand-alone distillation method and a strategy to postpone augmentation for better performance.

Findings

Postponing augmentation enhances RL training efficiency.

Adaptive scheduling improves generalization without extra samples.

The framework outperforms fixed augmentation schedules.

Abstract

In deep reinforcement learning (RL), data augmentation is widely considered as a tool to induce a set of useful priors about semantic consistency and improve sample efficiency and generalization performance. However, even when the prior is useful for generalization, distilling it to RL agent often interferes with RL training and degenerates sample efficiency. Meanwhile, the agent is forgetful of the prior due to the non-stationary nature of RL. These observations suggest two extreme schedules of distillation: (i) over the entire training; or (ii) only at the end. Hence, we devise a stand-alone network distillation method to inject the consistency prior at any time (even after RL), and a simple yet efficient framework to automatically schedule the distillation. Specifically, the proposed framework first focuses on mastering train environments regardless of generalization by adaptively deciding which {\it or no} augmentation to be used for the training. After this, we add the distillation to extract the remaining benefits for generalization from all the augmentations, which requires no additional new samples. In our experiments, we demonstrate the utility of the proposed framework, in particular, that considers postponing the augmentation to the end of RL training.