Instant Learning: Parallel Deep Neural Networks and Convolutional Bootstrapping

TL;DR

This paper introduces a parallel training method for deep neural networks using replication and convolutional bootstrapping, achieving rapid learning that surpasses traditional training after just one iteration.

Contribution

It proposes a novel parallel training framework for DNNs combined with convolutional bootstrapping, enabling faster and more robust learning.

Findings

Outperforms traditional DNN after one training iteration

Enables effective parallelization of DNN training

Enhances performance with convolutional bootstrapping

Abstract

Although deep neural networks (DNN) are able to scale with direct advances in computational power (e.g., memory and processing speed), they are not well suited to exploit the recent trends for parallel architectures. In particular, gradient descent is a sequential process and the resulting serial dependencies mean that DNN training cannot be parallelized effectively. Here, we show that a DNN may be replicated over a massive parallel architecture and used to provide a cumulative sampling of local solution space which results in rapid and robust learning. We introduce a complimentary convolutional bootstrapping approach that enhances performance of the parallel architecture further. Our parallelized convolutional bootstrapping DNN out-performs an identical fully-trained traditional DNN after only a single iteration of training.