Parle: parallelizing stochastic gradient descent

TL;DR

Parle is a parallel training algorithm for deep networks that accelerates convergence 2-4 times faster than traditional data-parallel SGD, improves error rates, and leverages flat minima phenomena without extra hyper-parameters.

Contribution

It introduces Parle, a novel parallel training method that reduces communication, speeds up convergence, and enhances generalization by exploiting flat minima.

Findings

Converges 2-4x faster than standard SGD.

Achieves near state-of-the-art error rates on CIFAR-10 and CIFAR-100.

Requires infrequent communication, suitable for distributed and multi-GPU setups.

Abstract

We propose a new algorithm called Parle for parallel training of deep networks that converges 2-4x faster than a data-parallel implementation of SGD, while achieving significantly improved error rates that are nearly state-of-the-art on several benchmarks including CIFAR-10 and CIFAR-100, without introducing any additional hyper-parameters. We exploit the phenomenon of flat minima that has been shown to lead to improved generalization error for deep networks. Parle requires very infrequent communication with the parameter server and instead performs more computation on each client, which makes it well-suited to both single-machine, multi-GPU settings and distributed implementations.