ResIST: Layer-Wise Decomposition of ResNets for Distributed Training

TL;DR

ResIST introduces a distributed training method for ResNets that decomposes the network into shallow sub-networks, reducing communication, memory, and time costs while maintaining competitive performance.

Contribution

It presents a novel layer-wise decomposition approach for ResNet training that minimizes communication and computation overheads in distributed settings.

Findings

Reduces communication and memory requirements significantly.

Maintains competitive model accuracy compared to traditional methods.

Efficiently trains ResNets with less resource consumption.

Abstract

We propose ResIST, a novel distributed training protocol for Residual Networks (ResNets). ResIST randomly decomposes a global ResNet into several shallow sub-ResNets that are trained independently in a distributed manner for several local iterations, before having their updates synchronized and aggregated into the global model. In the next round, new sub-ResNets are randomly generated and the process repeats until convergence. By construction, per iteration, ResIST communicates only a small portion of network parameters to each machine and never uses the full model during training. Thus, ResIST reduces the per-iteration communication, memory, and time requirements of ResNet training to only a fraction of the requirements of full-model training. In comparison to common protocols, like data-parallel training and data-parallel training with local SGD, ResIST yields a decrease in communication and compute requirements, while being competitive with respect to model performance.