StochasticNet: Forming Deep Neural Networks via Stochastic Connectivity

TL;DR

This paper introduces StochasticNet, a neural network architecture inspired by biological synaptic randomness, which uses stochastic connectivity to reduce complexity and improve efficiency while maintaining or enhancing accuracy across multiple datasets.

Contribution

The paper proposes a novel stochastic connectivity approach for deep neural networks, inspired by biological neural microcircuits, enabling reduced connections and improved performance.

Findings

Achieves comparable accuracy with less than half the neural connections.

Reduces overfitting on multiple datasets.

Faster operational speeds with maintained or improved accuracy.

Abstract

Deep neural networks is a branch in machine learning that has seen a meteoric rise in popularity due to its powerful abilities to represent and model high-level abstractions in highly complex data. One area in deep neural networks that is ripe for exploration is neural connectivity formation. A pivotal study on the brain tissue of rats found that synaptic formation for specific functional connectivity in neocortical neural microcircuits can be surprisingly well modeled and predicted as a random formation. Motivated by this intriguing finding, we introduce the concept of StochasticNet, where deep neural networks are formed via stochastic connectivity between neurons. As a result, any type of deep neural networks can be formed as a StochasticNet by allowing the neuron connectivity to be stochastic. Stochastic synaptic formations, in a deep neural network architecture, can allow for efficient utilization of neurons for performing specific tasks. To evaluate the feasibility of such a deep neural network architecture, we train a StochasticNet using four different image datasets (CIFAR-10, MNIST, SVHN, and STL-10). Experimental results show that a StochasticNet, using less than half the number of neural connections as a conventional deep neural network, achieves comparable accuracy and reduces overfitting on the CIFAR-10, MNIST and SVHN dataset. Interestingly, StochasticNet with less than half the number of neural connections, achieved a higher accuracy (relative improvement in test error rate of ~6% compared to ConvNet) on the STL-10 dataset than a conventional deep neural network. Finally, StochasticNets have faster operational speeds while achieving better or similar accuracy performances.