One weird trick for parallelizing convolutional neural networks

TL;DR

This paper introduces a novel parallelization technique for training convolutional neural networks across multiple GPUs, achieving superior scalability compared to existing methods.

Contribution

The paper proposes a new parallelization approach that significantly improves scalability for training modern CNNs on multiple GPUs.

Findings

Scales better than all existing methods

Effective for modern CNN architectures

Reduces training time with multiple GPUs

Abstract

I present a new way to parallelize the training of convolutional neural networks across multiple GPUs. The method scales significantly better than all alternatives when applied to modern convolutional neural networks.