Self Expanding Convolutional Neural Networks

TL;DR

This paper introduces a dynamic, self-expanding CNN method that adjusts model complexity during training, reducing resource use and improving performance, with validation on CIFAR-10 showing its effectiveness.

Contribution

It presents a novel self-expanding CNN approach that dynamically adjusts network size during training, enhancing efficiency and environmental sustainability.

Findings

Improved CNN performance with dynamic expansion on CIFAR-10.

Reduced computational resources and energy consumption.

Enabled extraction of multiple model checkpoints from a single training session.

Abstract

In this paper, we present a novel method for dynamically expanding Convolutional Neural Networks (CNNs) during training, aimed at meeting the increasing demand for efficient and sustainable deep learning models. Our approach, drawing from the seminal work on Self-Expanding Neural Networks (SENN), employs a natural expansion score as an expansion criteria to address the common issue of over-parameterization in deep convolutional neural networks, thereby ensuring that the model's complexity is finely tuned to the task's specific needs. A significant benefit of this method is its eco-friendly nature, as it obviates the necessity of training multiple models of different sizes. We employ a strategy where a single model is dynamically expanded, facilitating the extraction of checkpoints at various complexity levels, effectively reducing computational resource use and energy consumption while also expediting the development cycle by offering diverse model complexities from a single training session. We evaluate our method on the CIFAR-10 dataset and our experimental results validate this approach, demonstrating that dynamically adding layers not only maintains but also improves CNN performance, underscoring the effectiveness of our expansion criteria. This approach marks a considerable advancement in developing adaptive, scalable, and environmentally considerate neural network architectures, addressing key challenges in the field of deep learning.