Growing with Experience: Growing Neural Networks in Deep Reinforcement Learning

TL;DR

GrowNN introduces a progressive network growth method for deep reinforcement learning, enabling larger, more expressive policies without sacrificing trainability, leading to significant performance improvements in complex environments.

Contribution

The paper presents GrowNN, a novel approach that incrementally increases network capacity during training, enhancing policy expressiveness in RL without disrupting learning stability.

Findings

GrowNN outperforms static networks by up to 72% in performance.

Incremental growth improves policy expressiveness and training stability.

Applicable to various RL agents and environments.

Abstract

While increasingly large models have revolutionized much of the machine learning landscape, training even mid-sized networks for Reinforcement Learning (RL) is still proving to be a struggle. This, however, severely limits the complexity of policies we are able to learn. To enable increased network capacity while maintaining network trainability, we propose GrowNN, a simple yet effective method that utilizes progressive network growth during training. We start training a small network to learn an initial policy. Then we add layers without changing the encoded function. Subsequent updates can utilize the added layers to learn a more expressive policy, adding capacity as the policy's complexity increases. GrowNN can be seamlessly integrated into most existing RL agents. Our experiments on MiniHack and Mujoco show improved agent performance, with incrementally GrowNN-deeper networks outperforming their respective static counterparts of the same size by up to 48% on MiniHack Room and 72% on Ant.