Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Life

TL;DR

This paper introduces Synaptic Stripping, a method inspired by neuroscience, to remove problematic connections during training, thereby reviving dead ReLU neurons, improving network efficiency, and enhancing model capacity.

Contribution

The paper proposes a novel technique called Synaptic Stripping to automatically prune connections and revive dead neurons in neural networks, inspired by neurological processes.

Findings

Revives dead neurons, increasing model capacity.

Produces sparse, more efficient networks.

Improves performance on benchmark datasets with Vision Transformers.

Abstract

Rectified Linear Units (ReLU) are the default choice for activation functions in deep neural networks. While they demonstrate excellent empirical performance, ReLU activations can fall victim to the dead neuron problem. In these cases, the weights feeding into a neuron end up being pushed into a state where the neuron outputs zero for all inputs. Consequently, the gradient is also zero for all inputs, which means that the weights which feed into the neuron cannot update. The neuron is not able to recover from direct back propagation and model capacity is reduced as those parameters can no longer be further optimized. Inspired by a neurological process of the same name, we introduce Synaptic Stripping as a means to combat this dead neuron problem. By automatically removing problematic connections during training, we can regenerate dead neurons and significantly improve model capacity and parametric utilization. Synaptic Stripping is easy to implement and results in sparse networks that are more efficient than the dense networks they are derived from. We conduct several ablation studies to investigate these dynamics as a function of network width and depth and we conduct an exploration of Synaptic Stripping with Vision Transformers on a variety of benchmark datasets.