Decorrelated Soft Actor-Critic for Efficient Deep Reinforcement Learning

TL;DR

This paper introduces a novel online decorrelation method integrated into deep RL training, enhancing sample efficiency and training speed by improving credit assignment, demonstrated through the DSAC approach on Atari benchmarks.

Contribution

The paper proposes a new online decorrelation technique for deep RL that is seamlessly integrated into the training process, improving efficiency and performance.

Findings

Faster training in 5 out of 7 Atari games

Approximately 50% reduction in wall-clock time

Maintained performance levels across tested games

Abstract

The effectiveness of credit assignment in reinforcement learning (RL) when dealing with high-dimensional data is influenced by the success of representation learning via deep neural networks, and has implications for the sample efficiency of deep RL algorithms. Input decorrelation has been previously introduced as a method to speed up optimization in neural networks, and has proven impactful in both efficient deep learning and as a method for effective representation learning for deep RL algorithms. We propose a novel approach to online decorrelation in deep RL based on the decorrelated backpropagation algorithm that seamlessly integrates the decorrelation process into the RL training pipeline. Decorrelation matrices are added to each layer, which are updated using a separate decorrelation learning rule that minimizes the total decorrelation loss across all layers, in parallel to minimizing the usual RL loss. We used our approach in combination with the soft actor-critic (SAC) method, which we refer to as decorrelated soft actor-critic (DSAC). Experiments on the Atari 100k benchmark with DSAC shows, compared to the regular SAC baseline, faster training in five out of the seven games tested and improved reward performance in two games with around 50% reduction in wall-clock time, while maintaining performance levels on the other games. These results demonstrate the positive impact of network-wide decorrelation in deep RL for speeding up its sample efficiency through more effective credit assignment.