8-Bit Approximations for Parallelism in Deep Learning

TL;DR

This paper introduces 8-bit approximation algorithms for deep learning parallelism, significantly reducing communication bandwidth and achieving substantial speedups without sacrificing model accuracy.

Contribution

The paper develops and validates 8-bit gradient and activation approximations that enable faster parallel training of deep neural networks on large GPU systems.

Findings

2x data transfer speedup over 32-bit methods

50x speedup on 96 GPUs compared to 23x for 32-bit

Maintains predictive performance on multiple datasets

Abstract

The creation of practical deep learning data-products often requires parallelization across processors and computers to make deep learning feasible on large data sets, but bottlenecks in communication bandwidth make it difficult to attain good speedups through parallelism. Here we develop and test 8-bit approximation algorithms which make better use of the available bandwidth by compressing 32-bit gradients and nonlinear activations to 8-bit approximations. We show that these approximations do not decrease predictive performance on MNIST, CIFAR10, and ImageNet for both model and data parallelism and provide a data transfer speedup of 2x relative to 32-bit parallelism. We build a predictive model for speedups based on our experimental data, verify its validity on known speedup data, and show that we can obtain a speedup of 50x and more on a system of 96 GPUs compared to a speedup of 23x for 32-bit. We compare our data types with other methods and show that 8-bit approximations achieve state-of-the-art speedups for model parallelism. Thus 8-bit approximation is an efficient method to parallelize convolutional networks on very large systems of GPUs.