SCPL: Enhancing Neural Network Training Throughput with Decoupled Local Losses and Model Parallelism

TL;DR

SCPL introduces a novel training approach that decouples backpropagation, enabling parallel gradient computation across layers, significantly improving training throughput for large neural networks.

Contribution

The paper proposes Supervised Contrastive Parallel Learning (SCPL), a new method that decouples backpropagation to enhance model parallelism and training efficiency.

Findings

SCPL outperforms BP, Early Exit, GPipe, and AL in training efficiency.

SCPL reduces training time for large models.

Experimental results validate SCPL's effectiveness.

Abstract

Adopting large-scale AI models in enterprise information systems is often hindered by high training costs and long development cycles, posing a significant managerial challenge. The standard end-to-end backpropagation (BP) algorithm is a primary driver of modern AI, but it is also the source of inefficiency in training deep networks. This paper introduces a new training methodology, Supervised Contrastive Parallel Learning (SCPL), that addresses this issue by decoupling BP and transforming a long gradient flow into multiple short ones. This design enables the simultaneous computation of parameter gradients in different layers, achieving superior model parallelism and enhancing training throughput. Detailed experiments are presented to demonstrate the efficiency and effectiveness of our model compared to BP, Early Exit, GPipe, and Associated Learning (AL), a state-of-the-art method for decoupling backpropagation. By mitigating a fundamental performance bottleneck, SCPL provides a practical pathway for organizations to develop and deploy advanced information systems more cost-effectively and with greater agility. The experimental code is released for reproducibility. https://github.com/minyaho/scpl/