Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learning

TL;DR

This paper introduces Task-aware Lipschitz Data Augmentation (TLDA), a novel method for visual reinforcement learning that selectively augments task-irrelevant pixels to improve generalization and sample efficiency.

Contribution

The paper proposes TLDA, which identifies task-relevant pixels using Lipschitz constants and applies augmentation only to irrelevant pixels, enhancing RL training stability and performance.

Findings

TLDA outperforms previous methods on DeepMind Control, CARLA, and DeepMind Manipulation tasks.

TLDA improves sample efficiency and generalization in visual RL.

Selective augmentation of task-irrelevant pixels stabilizes training and boosts performance.

Abstract

One of the key challenges in visual Reinforcement Learning (RL) is to learn policies that can generalize to unseen environments. Recently, data augmentation techniques aiming at enhancing data diversity have demonstrated proven performance in improving the generalization ability of learned policies. However, due to the sensitivity of RL training, naively applying data augmentation, which transforms each pixel in a task-agnostic manner, may suffer from instability and damage the sample efficiency, thus further exacerbating the generalization performance. At the heart of this phenomenon is the diverged action distribution and high-variance value estimation in the face of augmented images. To alleviate this issue, we propose Task-aware Lipschitz Data Augmentation (TLDA) for visual RL, which explicitly identifies the task-correlated pixels with large Lipschitz constants, and only augments the task-irrelevant pixels. To verify the effectiveness of TLDA, we conduct extensive experiments on DeepMind Control suite, CARLA and DeepMind Manipulation tasks, showing that TLDA improves both sample efficiency in training time and generalization in test time. It outperforms previous state-of-the-art methods across the 3 different visual control benchmarks.