rQdia: Regularizing Q-Value Distributions With Image Augmentation

TL;DR

rQdia introduces a regularization method using image augmentation in pixel-based deep reinforcement learning, improving performance and sample efficiency across multiple benchmarks and tasks.

Contribution

It presents a novel auxiliary loss that regularizes Q-value distributions via image augmentation, enhancing existing algorithms like DrQ, SAC, and Data-Efficient Rainbow.

Findings

Boosts DrQ and SAC performance on most MuJoCo tasks.

Improves Data-Efficient Rainbow on many Atari environments.

Enables model-free continuous control directly from pixel inputs.

Abstract

rQdia regularizes Q-value distributions with augmented images in pixel-based deep reinforcement learning. With a simple auxiliary loss, that equalizes these distributions via MSE, rQdia boosts DrQ and SAC on 9/12 and 10/12 tasks respectively in the MuJoCo Continuous Control Suite from pixels, and Data-Efficient Rainbow on 18/26 Atari Arcade environments. Gains are measured in both sample efficiency and longer-term training. Moreover, the addition of rQdia finally propels model-free continuous control from pixels over the state encoding baseline.