Synkhronos: a Multi-GPU Theano Extension for Data Parallelism

TL;DR

Synkhronos is a Theano extension enabling efficient multi-GPU data parallelism with automated synchronization, achieving near-linear speedup in deep learning training tasks like ResNet-50.

Contribution

It introduces Synkhronos, a novel framework that simplifies multi-GPU programming in Theano, providing automation and high performance for data-parallel computations.

Findings

Achieves 7.5x speedup on 8 GPUs for ResNet-50 training.

Provides automated synchronization and data-parallel execution in Theano.

Enhances Theano with input slicing and indexing for efficient parallelism.

Abstract

We present Synkhronos, an extension to Theano for multi-GPU computations leveraging data parallelism. Our framework provides automated execution and synchronization across devices, allowing users to continue to write serial programs without risk of race conditions. The NVIDIA Collective Communication Library is used for high-bandwidth inter-GPU communication. Further enhancements to the Theano function interface include input slicing (with aggregation) and input indexing, which perform common data-parallel computation patterns efficiently. One example use case is synchronous SGD, which has recently been shown to scale well for a growing set of deep learning problems. When training ResNet-50, we achieve a near-linear speedup of 7.5x on an NVIDIA DGX-1 using 8 GPUs, relative to Theano-only code running a single GPU in isolation. Yet Synkhronos remains general to any data-parallel computation programmable in Theano. By implementing parallelism at the level of individual Theano functions, our framework uniquely addresses a niche between manual multi-device programming and prescribed multi-GPU training routines.