Optimal Policy Sparsification and Low Rank Decomposition for Deep Reinforcement Learning

TL;DR

This paper introduces a novel $L_0$-norm regularization method for deep reinforcement learning that sparsifies policies and enables low-rank decomposition, significantly reducing resource consumption without performance loss.

Contribution

The authors propose a new $L_0$-norm regularization technique for DRL policies that achieves high sparsity and effective low-rank decomposition, outperforming existing methods.

Findings

Achieved 93% sparsity and 70% compression in SuperMarioBros environment.

Attained 36% sparsity and 46% compression in Surgical Robot Learning environment.

Demonstrated improved performance with minimal reward decay using the proposed method.

Abstract

Deep reinforcement learning(DRL) has shown significant promise in a wide range of applications including computer games and robotics. Yet, training DRL policies consume extraordinary computing resources resulting in dense policies which are prone to overfitting. Moreover, inference with dense DRL policies limit their practical applications, especially in edge computing. Techniques such as pruning and singular value decomposition have been used with deep learning models to achieve sparsification and model compression to limit overfitting and reduce memory consumption. However, these techniques resulted in sub-optimal performance with notable decay in rewards. $L_1$ and $L_2$ regularization techniques have been proposed for neural network sparsification and sparse auto-encoder development, but their implementation in DRL environments has not been apparent. We propose a novel $L_0$-norm-regularization technique using an optimal sparsity map to sparsify DRL policies and promote their decomposition to a lower rank without decay in rewards. We evaluated our $L_0$-norm-regularization technique across five different environments (Cartpole-v1, Acrobat-v1, LunarLander-v2, SuperMarioBros-7.1.v0 and Surgical Robot Learning) using several on-policy and off-policy algorithms. We demonstrated that the $L_0$-norm-regularized DRL policy in the SuperMarioBros environment achieved 93% sparsity and gained 70% compression when subjected to low-rank decomposition, while significantly outperforming the dense policy. Additionally, the $L_0$-norm-regularized DRL policy in the Surgical Robot Learning environment achieved a 36% sparsification and gained 46% compression when decomposed to a lower rank, while being performant. The results suggest that our custom $L_0$-norm-regularization technique for sparsification of DRL policies is a promising avenue to reduce computational resources and limit overfitting.