Improving Neural Network Generalization by Combining Parallel Circuits with Dropout

TL;DR

This paper introduces an improved neural network architecture that combines Parallel Circuits with Dropout, achieving faster training times while maintaining or improving generalization performance.

Contribution

It extends Dropout to Parallel Circuits, providing a novel approach that balances speed and generalization in neural network training.

Findings

Improved error rates in most experiments

Maintains speed advantages of Parallel Circuits

Provides insights into fusion approaches

Abstract

In an attempt to solve the lengthy training times of neural networks, we proposed Parallel Circuits (PCs), a biologically inspired architecture. Previous work has shown that this approach fails to maintain generalization performance in spite of achieving sharp speed gains. To address this issue, and motivated by the way Dropout prevents node co-adaption, in this paper, we suggest an improvement by extending Dropout to the PC architecture. The paper provides multiple insights into this combination, including a variety of fusion approaches. Experiments show promising results in which improved error rates are achieved in most cases, whilst maintaining the speed advantage of the PC approach.