Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization

TL;DR

This paper enhances off-policy reinforcement learning by integrating weight normalization into CrossQ, enabling stable scaling with higher update-to-data ratios and improving sample efficiency across complex continuous control tasks.

Contribution

It introduces weight normalization into CrossQ, allowing stable scaling with higher UTD ratios and eliminating the need for network resets in reinforcement learning.

Findings

Achieves competitive performance on DeepMind Control Suite and Myosuite benchmarks.

Successfully scales with higher UTD ratios without training instability.

Improves sample efficiency in complex continuous control environments.

Abstract

Reinforcement learning has achieved significant milestones, but sample efficiency remains a bottleneck for real-world applications. Recently, CrossQ has demonstrated state-of-the-art sample efficiency with a low update-to-data (UTD) ratio of 1. In this work, we explore CrossQ's scaling behavior with higher UTD ratios. We identify challenges in the training dynamics, which are emphasized by higher UTD ratios. To address these, we integrate weight normalization into the CrossQ framework, a solution that stabilizes training, has been shown to prevent potential loss of plasticity and keeps the effective learning rate constant. Our proposed approach reliably scales with increasing UTD ratios, achieving competitive performance across 25 challenging continuous control tasks on the DeepMind Control Suite and Myosuite benchmarks, notably the complex dog and humanoid environments. This work eliminates the need for drastic interventions, such as network resets, and offers a simple yet robust pathway for improving sample efficiency and scalability in model-free reinforcement learning.