Don't Use Large Mini-Batches, Use Local SGD

TL;DR

This paper introduces post-local SGD, a method that enhances the generalization of large-batch training in deep neural networks without sacrificing efficiency or scalability.

Contribution

It proposes a novel post-local SGD approach that improves generalization in large-batch training and provides a comprehensive analysis of local SGD variants.

Findings

Post-local SGD significantly improves generalization on standard benchmarks.

The method maintains efficiency and scalability comparable to large-batch training.

Extensive study of communication-performance trade-offs in local SGD variants.

Abstract

Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key efficiency and scalability gains in recent years. However, progress faces a major roadblock, as models trained with large batches often do not generalize well, i.e. they do not show good accuracy on new data. As a remedy, we propose a \emph{post-local} SGD and show that it significantly improves the generalization performance compared to large-batch training on standard benchmarks while enjoying the same efficiency (time-to-accuracy) and scalability. We further provide an extensive study of the communication efficiency vs. performance trade-offs associated with a host of \emph{local SGD} variants.