Exploring The Neural Burden In Pruned Models: An Insight Inspired By Neuroscience

TL;DR

This paper investigates the neural burden phenomenon in pruned neural networks, inspired by neuroscience, and proposes a mitigation method to improve performance of pruning-during-training models, supported by extensive experiments.

Contribution

It introduces the concept of neural burden in pruned models and offers a simple approach to reduce performance decline during pruning, applicable across various techniques.

Findings

Neural burden phenomenon exists in pruned models.

Proposed method effectively mitigates performance loss.

Experimental results demonstrate the potential of the approach.

Abstract

Vision Transformer and its variants have been adopted in many visual tasks due to their powerful capabilities, which also bring significant challenges in computation and storage. Consequently, researchers have introduced various compression methods in recent years, among which the pruning techniques are widely used to remove a significant fraction of the network. Therefore, these methods can reduce significant percent of the FLOPs, but often lead to a decrease in model performance. To investigate the underlying causes, we focus on the pruning methods specifically belonging to the pruning-during-training category, then drew inspiration from neuroscience and propose a new concept for artificial neural network models named Neural Burden. We investigate its impact in the model pruning process, and subsequently explore a simple yet effective approach to mitigate the decline in model performance, which can be applied to any pruning-during-training technique. Extensive experiments indicate that the neural burden phenomenon indeed exists, and show the potential of our method. We hope that our findings can provide valuable insights for future research. Code will be made publicly available after this paper is published.