Parareal Neural Networks Emulating a Parallel-in-time Algorithm

TL;DR

This paper introduces a novel method to construct parallel neural networks inspired by the parareal parallel-in-time algorithm, enabling multi-GPU training acceleration while maintaining accuracy.

Contribution

It proposes a new approach to emulate the parareal algorithm within DNNs, allowing layers to be parallelized across GPUs for faster training.

Findings

Achieved accelerated training on VGG-16 and ResNet-1001

Maintained accuracy comparable to traditional training methods

Demonstrated effectiveness across multiple datasets

Abstract

As deep neural networks (DNNs) become deeper, the training time increases. In this perspective, multi-GPU parallel computing has become a key tool in accelerating the training of DNNs. In this paper, we introduce a novel methodology to construct a parallel neural network that can utilize multiple GPUs simultaneously from a given DNN. We observe that layers of DNN can be interpreted as the time step of a time-dependent problem and can be parallelized by emulating a parallel-in-time algorithm called parareal. The parareal algorithm consists of fine structures which can be implemented in parallel and a coarse structure which gives suitable approximations to the fine structures. By emulating it, the layers of DNN are torn to form a parallel structure which is connected using a suitable coarse network. We report accelerated and accuracy-preserved results of the proposed methodology applied to VGG-16 and ResNet-1001 on several datasets.