Growing Deep Neural Network Considering with Similarity between Neurons

TL;DR

This paper introduces a novel neural network training method inspired by neurogenesis, progressively increasing neurons and applying similarity constraints to improve accuracy and interpretability in image recognition tasks.

Contribution

It proposes a new approach that dynamically grows neurons during training with similarity-based constraints, reducing redundancy and enhancing feature relevance.

Findings

Improved accuracy on CIFAR-10 and CIFAR-100 datasets.

Enhanced focus on whole objects in visual explanations.

Effective management of computational costs during training.

Abstract

Deep learning has excelled in image recognition tasks through neural networks inspired by the human brain. However, the necessity for large models to improve prediction accuracy introduces significant computational demands and extended training times.Conventional methods such as fine-tuning, knowledge distillation, and pruning have the limitations like potential accuracy drops. Drawing inspiration from human neurogenesis, where neuron formation continues into adulthood, we explore a novel approach of progressively increasing neuron numbers in compact models during training phases, thereby managing computational costs effectively. We propose a method that reduces feature extraction biases and neuronal redundancy by introducing constraints based on neuron similarity distributions. This approach not only fosters efficient learning in new neurons but also enhances feature extraction relevancy for given tasks. Results on CIFAR-10 and CIFAR-100 datasets demonstrated accuracy improvement, and our method pays more attention to whole object to be classified in comparison with conventional method through Grad-CAM visualizations. These results suggest that our method's potential to decision-making processes.