Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid

TL;DR

This paper introduces a multigrid-based algorithm for training deep residual neural networks that significantly accelerates GPU training by enabling layer-wise parallelism and concurrent execution, achieving over ten times faster performance.

Contribution

It presents a novel multigrid full approximation storage algorithm that allows for efficient layer-parallel training and GPU concurrency in deep residual networks.

Findings

Achieved 10.2x speedup over traditional methods

Enabled layer-wise parallelism and GPU concurrency

Demonstrated effectiveness on deep residual networks

Abstract

A Multigrid Full Approximation Storage algorithm for solving Deep Residual Networks is developed to enable neural network parallelized layer-wise training and concurrent computational kernel execution on GPUs. This work demonstrates a 10.2x speedup over traditional layer-wise model parallelism techniques using the same number of compute units.