RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch

TL;DR

RLx2 is a novel sparse deep reinforcement learning training framework that enables training from scratch with high compression rates and minimal performance loss, significantly reducing computational costs.

Contribution

The paper introduces RLx2, a new sparse DRL training method using gradient-based topology evolution and a multi-step TD target, enabling training entirely on sparse networks from scratch.

Findings

Achieves 7.5x-20x model compression with less than 3% performance degradation.

Reduces training FLOPs by up to 20x and inference FLOPs by 50x.

State-of-the-art sparse training performance in several DRL tasks.

Abstract

Training deep reinforcement learning (DRL) models usually requires high computation costs. Therefore, compressing DRL models possesses immense potential for training acceleration and model deployment. However, existing methods that generate small models mainly adopt the knowledge distillation-based approach by iteratively training a dense network. As a result, the training process still demands massive computing resources. Indeed, sparse training from scratch in DRL has not been well explored and is particularly challenging due to non-stationarity in bootstrap training. In this work, we propose a novel sparse DRL training framework, "the Rigged Reinforcement Learning Lottery" (RLx2), which builds upon gradient-based topology evolution and is capable of training a sparse DRL model based entirely on a sparse network. Specifically, RLx2 introduces a novel multi-step TD target mechanism with a dynamic-capacity replay buffer to achieve robust value learning and efficient topology exploration in sparse models. It also reaches state-of-the-art sparse training performance in several tasks, showing 7.5\times-20\times model compression with less than 3% performance degradation and up to 20\times and 50\times FLOPs reduction for training and inference, respectively.