Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learning

TL;DR

This paper introduces a novel layer-wise partitioning and merging framework for deep neural network training that enhances speed and scalability by reducing communication overhead and addressing locking issues, achieving near-linear speedup.

Contribution

It proposes a new layer-wise partitioning and merging method combined with parallel forward and backward passes to improve training efficiency and scalability.

Findings

Outperforms state-of-the-art methods in training speed

Achieves almost linear speedup without accuracy loss

Reduces communication overhead during training

Abstract

Deep Neural Network (DNN) models are usually trained sequentially from one layer to another, which causes forward, backward and update locking's problems, leading to poor performance in terms of training time. The existing parallel strategies to mitigate these problems provide suboptimal runtime performance. In this work, we have proposed a novel layer-wise partitioning and merging, forward and backward pass parallel framework to provide better training performance. The novelty of the proposed work consists of 1) a layer-wise partition and merging model which can minimise communication overhead between devices without the memory cost of existing strategies during the training process; 2) a forward pass and backward pass parallelisation and optimisation to address the update locking problem and minimise the total training cost. The experimental evaluation on real use cases shows that the proposed method outperforms the state-of-the-art approaches in terms of training speed; and achieves almost linear speedup without compromising the accuracy performance of the non-parallel approach.