Interlocking Backpropagation: Improving depthwise model-parallelism

TL;DR

This paper introduces interlocking backpropagation, a novel training strategy that enhances resource utilization and task performance in large-scale distributed neural network training, bridging the gap between local and global learning.

Contribution

It proposes interlocking backpropagation, a new class of strategies that improve resource efficiency and task performance in model-parallel training of neural networks.

Findings

Outperforms local learning in task accuracy.

Outperforms global learning in training efficiency.

Effective on ResNets and Transformer models.

Abstract

The number of parameters in state of the art neural networks has drastically increased in recent years. This surge of interest in large scale neural networks has motivated the development of new distributed training strategies enabling such models. One such strategy is model-parallel distributed training. Unfortunately, model-parallelism can suffer from poor resource utilisation, which leads to wasted resources. In this work, we improve upon recent developments in an idealised model-parallel optimisation setting: local learning. Motivated by poor resource utilisation in the global setting and poor task performance in the local setting, we introduce a class of intermediary strategies between local and global learning referred to as interlocking backpropagation. These strategies preserve many of the compute-efficiency advantages of local optimisation, while recovering much of the task performance achieved by global optimisation. We assess our strategies on both image classification ResNets and Transformer language models, finding that our strategy consistently out-performs local learning in terms of task performance, and out-performs global learning in training efficiency.