Gear Training: A new way to implement high-performance model-parallel training

TL;DR

This paper introduces Gear Training, a novel model-parallel training method that splits deep neural networks into parts and trains them at different speeds to improve efficiency on large clusters.

Contribution

It proposes a new parallel training approach that differs from existing methods by allowing different parts of the model to be trained at varying speeds.

Findings

Demonstrates improved training efficiency on large clusters

Reduces synchronization overhead compared to traditional methods

Enables scalable training of very deep neural networks

Abstract

The training of Deep Neural Networks usually needs tremendous computing resources. Therefore many deep models are trained in large cluster instead of single machine or GPU. Though major researchs at present try to run whole model on all machines by using asynchronous asynchronous stochastic gradient descent (ASGD), we present a new approach to train deep model parallely -- split the model and then seperately train different parts of it in different speed.