Leapfrogging for parallelism in deep neural networks

TL;DR

The paper introduces leapfrogging, a parallelization technique for backpropagation in deep neural networks, significantly reducing computation time proportional to the number of processing units.

Contribution

Leapfrogging is a novel method that enables efficient parallelization of backpropagation, improving training speed in deep neural networks.

Findings

Reduces backpropagation computation by a factor of 1-1/k

Demonstrates effective parallelization with multiple GPUs

Provides theoretical analysis of speedup benefits

Abstract

We present a technique, which we term leapfrogging, to parallelize back- propagation in deep neural networks. We show that this technique yields a savings of $1-1/k$ of a dominant term in backpropagation, where k is the number of threads (or gpus).