# Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks

## TL;DR

This paper reveals that training deep neural networks with ReLU, L2 regularization, and Adam optimizer naturally induces weight sparsity, enabling effective model reduction without additional regularizers.

## Contribution

The study identifies conditions under which Adam training induces implicit weight sparsity and proposes a simple method to reduce model size by removing zero weights.

## Key findings

- Adam training induces group sparsity in weights.
- The proposed reduction method maintains performance.
- Effective size reduction demonstrated on MNIST and CIFAR-10.

## Abstract

In recent years, deep neural networks (DNNs) have been applied to various machine leaning tasks, including image recognition, speech recognition, and machine translation. However, large DNN models are needed to achieve state-of-the-art performance, exceeding the capabilities of edge devices. Model reduction is thus needed for practical use. In this paper, we point out that deep learning automatically induces group sparsity of weights, in which all weights connected to an output channel (node) are zero, when training DNNs under the following three conditions: (1) rectified-linear-unit (ReLU) activations, (2) an $L_2$-regularized objective function, and (3) the Adam optimizer. Next, we analyze this behavior both theoretically and experimentally, and propose a simple model reduction method: eliminate the zero weights after training the DNN. In experiments on MNIST and CIFAR-10 datasets, we demonstrate the sparsity with various training setups. Finally, we show that our method can efficiently reduce the model size and performs well relative to methods that use a sparsity-inducing regularizer.